2024-09-13T13:43:48.225 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T13:43:48.237 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T13:43:48.320 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903521 branch: wip-aclamk-cbt-combined description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} email: akupczyk@redhat.com first_in_suite: false job_id: '7903521' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-aclamk-cbt-combined ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 workunit: branch: wip-aclamk-cbt-combined sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 owner: scheduled_akupczyk@teuthology priority: 160 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 4159 sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 sleep_before_teardown: 0 subset: 682/13280 suite: rados suite_branch: wip-aclamk-cbt-combined suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 targets: smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKzKms5ZjcWC0AA37jGuH8EvaMlq+zrgSWDqSUEptwTTew2b8riCVtaAamr0k4LbOrbHSA+SRJb8fWIuPIaZ3mQ= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_09:43:51 tube: smithi user: akupczyk verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T13:43:48.320 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa; will attempt to use it 2024-09-13T13:43:48.321 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_d58f73069a8157a9ace12a0247ee3e8497707ea9/qa/tasks 2024-09-13T13:43:48.321 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T13:43:48.323 INFO:teuthology.task.internal:Checking packages... 2024-09-13T13:43:48.347 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd58f73069a8157a9ace12a0247ee3e8497707ea9' 2024-09-13T13:43:48.347 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T13:43:48.347 INFO:teuthology.packaging:ref: None 2024-09-13T13:43:48.348 INFO:teuthology.packaging:tag: None 2024-09-13T13:43:48.348 INFO:teuthology.packaging:branch: wip-aclamk-cbt-combined 2024-09-13T13:43:48.348 INFO:teuthology.packaging:sha1: d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:43:48.348 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-aclamk-cbt-combined 2024-09-13T13:43:48.502 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4697-gd58f7306-1jammy 2024-09-13T13:43:48.503 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T13:43:48.510 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T13:43:48.510 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T13:43:48.562 INFO:teuthology.task.internal:Saving configuration 2024-09-13T13:43:48.571 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T13:43:48.578 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T13:43:48.611 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903521', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 13:38:18.197146', 'locked_by': 'scheduled_akupczyk@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIE/OICMKolSmGocJqIRUpReFit8EUFgmNIbMWByQtzq'} 2024-09-13T13:43:48.611 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T13:43:48.617 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-13T13:43:48.617 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T13:43:48.670 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f02a2a6fbe0>, signals=[15]) 2024-09-13T13:43:48.670 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T13:43:48.677 INFO:teuthology.task.internal:Opening connections... 2024-09-13T13:43:48.677 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2024-09-13T13:43:48.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:43:48.766 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T13:43:48.772 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2024-09-13T13:43:48.869 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2024-09-13T13:43:48.870 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2024-09-13T13:43:48.916 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-13T13:43:48.916 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="Ubuntu" 2024-09-13T13:43:48.917 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="22.04" 2024-09-13T13:43:48.917 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-13T13:43:48.917 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_CODENAME=jammy 2024-09-13T13:43:48.917 INFO:teuthology.orchestra.run.smithi112.stdout:ID=ubuntu 2024-09-13T13:43:48.917 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE=debian 2024-09-13T13:43:48.917 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-13T13:43:48.917 INFO:teuthology.orchestra.run.smithi112.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-13T13:43:48.917 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-13T13:43:48.917 INFO:teuthology.orchestra.run.smithi112.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-13T13:43:48.917 INFO:teuthology.orchestra.run.smithi112.stdout:UBUNTU_CODENAME=jammy 2024-09-13T13:43:48.918 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2024-09-13T13:43:48.948 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T13:43:48.956 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T13:43:48.995 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T13:43:48.996 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2024-09-13T13:43:49.001 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T13:43:49.020 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T13:43:49.020 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2024-09-13T13:43:49.228 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T13:43:49.267 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T13:43:49.283 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T13:43:49.283 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T13:43:49.283 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T13:43:49.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:43:49.288 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-09-13T13:43:49.337 INFO:teuthology.orchestra.run.smithi112.stdout:5.15.0-56-generic 2024-09-13T13:43:49.337 INFO:teuthology.task.kernel:Running kernel on smithi112: 5.15.0-56-generic 2024-09-13T13:43:49.338 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get clean 2024-09-13T13:43:49.455 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get update 2024-09-13T13:43:49.580 INFO:teuthology.orchestra.run.smithi112.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-13T13:43:49.699 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T13:43:49.780 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-13T13:43:49.790 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-13T13:43:49.889 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-13T13:43:49.892 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-13T13:43:49.895 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-13T13:43:49.895 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-13T13:43:49.896 INFO:teuthology.orchestra.run.smithi112.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-13T13:43:49.917 INFO:teuthology.orchestra.run.smithi112.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-13T13:43:49.920 INFO:teuthology.orchestra.run.smithi112.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-13T13:43:49.920 INFO:teuthology.orchestra.run.smithi112.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-13T13:43:49.926 INFO:teuthology.orchestra.run.smithi112.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-13T13:43:49.931 INFO:teuthology.orchestra.run.smithi112.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-13T13:43:49.934 INFO:teuthology.orchestra.run.smithi112.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-13T13:43:49.934 INFO:teuthology.orchestra.run.smithi112.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-13T13:43:49.934 INFO:teuthology.orchestra.run.smithi112.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-13T13:43:49.935 INFO:teuthology.orchestra.run.smithi112.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-13T13:43:49.961 INFO:teuthology.orchestra.run.smithi112.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-13T13:43:50.135 INFO:teuthology.orchestra.run.smithi112.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-13T13:43:50.273 INFO:teuthology.orchestra.run.smithi112.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-13T13:43:50.591 INFO:teuthology.orchestra.run.smithi112.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-13T13:43:50.627 INFO:teuthology.orchestra.run.smithi112.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-13T13:43:50.641 INFO:teuthology.orchestra.run.smithi112.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-13T13:43:50.641 INFO:teuthology.orchestra.run.smithi112.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-13T13:43:50.711 INFO:teuthology.orchestra.run.smithi112.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-13T13:43:50.712 INFO:teuthology.orchestra.run.smithi112.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-13T13:43:50.728 INFO:teuthology.orchestra.run.smithi112.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-13T13:43:50.728 INFO:teuthology.orchestra.run.smithi112.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-13T13:43:50.750 INFO:teuthology.orchestra.run.smithi112.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-13T13:43:50.779 INFO:teuthology.orchestra.run.smithi112.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-13T13:43:50.785 INFO:teuthology.orchestra.run.smithi112.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-13T13:43:50.788 INFO:teuthology.orchestra.run.smithi112.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-13T13:43:50.788 INFO:teuthology.orchestra.run.smithi112.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-13T13:43:50.837 INFO:teuthology.orchestra.run.smithi112.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-13T13:43:50.837 INFO:teuthology.orchestra.run.smithi112.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-13T13:43:50.837 INFO:teuthology.orchestra.run.smithi112.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-13T13:43:50.839 INFO:teuthology.orchestra.run.smithi112.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-13T13:43:50.840 INFO:teuthology.orchestra.run.smithi112.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-13T13:43:50.918 INFO:teuthology.orchestra.run.smithi112.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-13T13:43:50.918 INFO:teuthology.orchestra.run.smithi112.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-13T13:43:50.919 INFO:teuthology.orchestra.run.smithi112.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-13T13:43:50.920 INFO:teuthology.orchestra.run.smithi112.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-13T13:43:50.920 INFO:teuthology.orchestra.run.smithi112.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-13T13:43:56.171 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 16.0 MB in 2s (7,520 kB/s) 2024-09-13T13:43:57.481 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-13T13:43:57.498 DEBUG:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T13:43:57.559 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-13T13:43:57.770 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2024-09-13T13:43:57.771 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2024-09-13T13:43:57.951 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T13:43:57.951 INFO:teuthology.orchestra.run.smithi112.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T13:43:57.951 INFO:teuthology.orchestra.run.smithi112.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T13:43:57.952 INFO:teuthology.orchestra.run.smithi112.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T13:43:57.952 INFO:teuthology.orchestra.run.smithi112.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-13T13:43:57.952 INFO:teuthology.orchestra.run.smithi112.stdout: python2.7-minimal 2024-09-13T13:43:57.952 INFO:teuthology.orchestra.run.smithi112.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T13:43:57.953 INFO:teuthology.orchestra.run.smithi112.stdout:The following additional packages will be installed: 2024-09-13T13:43:57.953 INFO:teuthology.orchestra.run.smithi112.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T13:43:57.953 INFO:teuthology.orchestra.run.smithi112.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-13T13:43:57.954 INFO:teuthology.orchestra.run.smithi112.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-13T13:43:57.955 INFO:teuthology.orchestra.run.smithi112.stdout:Suggested packages: 2024-09-13T13:43:57.955 INFO:teuthology.orchestra.run.smithi112.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-13T13:43:57.955 INFO:teuthology.orchestra.run.smithi112.stdout:Recommended packages: 2024-09-13T13:43:57.955 INFO:teuthology.orchestra.run.smithi112.stdout: thermald 2024-09-13T13:43:58.002 INFO:teuthology.orchestra.run.smithi112.stdout:The following NEW packages will be installed: 2024-09-13T13:43:58.002 INFO:teuthology.orchestra.run.smithi112.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-13T13:43:58.002 INFO:teuthology.orchestra.run.smithi112.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-13T13:43:58.002 INFO:teuthology.orchestra.run.smithi112.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-13T13:43:58.003 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages will be upgraded: 2024-09-13T13:43:58.004 INFO:teuthology.orchestra.run.smithi112.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-13T13:43:58.184 INFO:teuthology.orchestra.run.smithi112.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T13:43:58.185 INFO:teuthology.orchestra.run.smithi112.stdout:Need to get 113 MB of archives. 2024-09-13T13:43:58.185 INFO:teuthology.orchestra.run.smithi112.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-13T13:43:58.185 INFO:teuthology.orchestra.run.smithi112.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-13T13:43:59.667 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-13T13:44:00.014 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-13T13:44:02.207 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-13T13:44:02.249 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-13T13:44:02.249 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-13T13:44:02.591 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-13T13:44:02.838 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-13T13:44:03.233 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 113 MB in 5s (23.5 MB/s) 2024-09-13T13:44:03.404 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-13T13:44:05.345 INFO:teuthology.orchestra.run.smithi112.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-13T13:44:05.346 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T13:44:05.410 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T13:44:07.445 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-13T13:44:07.466 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T13:44:07.560 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T13:44:07.982 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-13T13:44:08.002 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T13:44:08.040 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T13:44:14.425 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T13:44:14.543 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T13:44:14.817 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T13:44:14.930 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T13:44:15.232 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-13T13:44:15.243 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-13T13:44:15.282 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T13:44:20.706 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-13T13:44:20.743 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-13T13:44:20.781 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T13:44:23.261 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-13T13:44:23.367 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-13T13:44:23.658 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-13T13:44:23.780 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T13:44:23.906 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-13T13:44:24.033 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T13:44:25.430 INFO:teuthology.orchestra.run.smithi112.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-13T13:44:25.430 INFO:teuthology.orchestra.run.smithi112.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-13T13:44:25.430 INFO:teuthology.orchestra.run.smithi112.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-13T13:44:25.430 INFO:teuthology.orchestra.run.smithi112.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-13T13:44:26.112 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T13:44:27.512 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T13:44:29.089 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-13T13:44:29.214 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-13T13:44:29.341 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-13T13:44:29.416 INFO:teuthology.orchestra.run.smithi112.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-13T13:44:29.416 INFO:teuthology.orchestra.run.smithi112.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-13T13:44:43.163 INFO:teuthology.orchestra.run.smithi112.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-13T13:44:43.164 INFO:teuthology.orchestra.run.smithi112.stdout:Sourcing file `/etc/default/grub' 2024-09-13T13:44:43.170 INFO:teuthology.orchestra.run.smithi112.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T13:44:43.197 INFO:teuthology.orchestra.run.smithi112.stdout:Generating grub configuration file ... 2024-09-13T13:44:43.441 INFO:teuthology.orchestra.run.smithi112.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T13:44:43.457 INFO:teuthology.orchestra.run.smithi112.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T13:44:43.727 INFO:teuthology.orchestra.run.smithi112.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T13:44:43.732 INFO:teuthology.orchestra.run.smithi112.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T13:44:43.792 INFO:teuthology.orchestra.run.smithi112.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T13:44:43.797 INFO:teuthology.orchestra.run.smithi112.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T13:44:43.940 INFO:teuthology.orchestra.run.smithi112.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T13:44:43.940 INFO:teuthology.orchestra.run.smithi112.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T13:44:43.940 INFO:teuthology.orchestra.run.smithi112.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T13:44:43.978 INFO:teuthology.orchestra.run.smithi112.stdout:done 2024-09-13T13:44:44.444 DEBUG:teuthology.orchestra.run.smithi112:> dpkg -s linux-image-generic 2024-09-13T13:44:44.469 INFO:teuthology.orchestra.run.smithi112.stdout:Package: linux-image-generic 2024-09-13T13:44:44.470 INFO:teuthology.orchestra.run.smithi112.stdout:Status: install ok installed 2024-09-13T13:44:44.470 INFO:teuthology.orchestra.run.smithi112.stdout:Priority: optional 2024-09-13T13:44:44.470 INFO:teuthology.orchestra.run.smithi112.stdout:Section: kernel 2024-09-13T13:44:44.470 INFO:teuthology.orchestra.run.smithi112.stdout:Installed-Size: 21 2024-09-13T13:44:44.470 INFO:teuthology.orchestra.run.smithi112.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T13:44:44.470 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: amd64 2024-09-13T13:44:44.470 INFO:teuthology.orchestra.run.smithi112.stdout:Source: linux-meta 2024-09-13T13:44:44.470 INFO:teuthology.orchestra.run.smithi112.stdout:Version: 5.15.0.121.121 2024-09-13T13:44:44.470 INFO:teuthology.orchestra.run.smithi112.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T13:44:44.470 INFO:teuthology.orchestra.run.smithi112.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T13:44:44.470 INFO:teuthology.orchestra.run.smithi112.stdout:Recommends: thermald 2024-09-13T13:44:44.471 INFO:teuthology.orchestra.run.smithi112.stdout:Description: Generic Linux kernel image 2024-09-13T13:44:44.471 INFO:teuthology.orchestra.run.smithi112.stdout: This package will always depend on the latest generic kernel image 2024-09-13T13:44:44.471 INFO:teuthology.orchestra.run.smithi112.stdout: available. 2024-09-13T13:44:44.471 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-13T13:44:44.471 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T13:44:44.471 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T13:44:44.471 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T13:44:44.471 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T13:44:44.472 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get clean 2024-09-13T13:44:44.644 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get update 2024-09-13T13:44:44.765 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T13:44:44.775 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T13:44:44.793 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T13:44:44.841 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-13T13:44:45.145 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 129 kB in 0s (373 kB/s) 2024-09-13T13:44:46.140 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-13T13:44:46.158 DEBUG:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T13:44:46.208 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-13T13:44:46.393 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2024-09-13T13:44:46.393 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2024-09-13T13:44:46.561 INFO:teuthology.orchestra.run.smithi112.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-13T13:44:46.561 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T13:44:46.561 INFO:teuthology.orchestra.run.smithi112.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T13:44:46.562 INFO:teuthology.orchestra.run.smithi112.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T13:44:46.562 INFO:teuthology.orchestra.run.smithi112.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T13:44:46.562 INFO:teuthology.orchestra.run.smithi112.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T13:44:46.562 INFO:teuthology.orchestra.run.smithi112.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T13:44:46.562 INFO:teuthology.orchestra.run.smithi112.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T13:44:46.601 INFO:teuthology.orchestra.run.smithi112.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-13T13:44:46.603 DEBUG:teuthology.orchestra.run.smithi112:> dpkg -s linux-image-generic 2024-09-13T13:44:46.662 INFO:teuthology.orchestra.run.smithi112.stdout:Package: linux-image-generic 2024-09-13T13:44:46.663 INFO:teuthology.orchestra.run.smithi112.stdout:Status: install ok installed 2024-09-13T13:44:46.663 INFO:teuthology.orchestra.run.smithi112.stdout:Priority: optional 2024-09-13T13:44:46.663 INFO:teuthology.orchestra.run.smithi112.stdout:Section: kernel 2024-09-13T13:44:46.663 INFO:teuthology.orchestra.run.smithi112.stdout:Installed-Size: 21 2024-09-13T13:44:46.663 INFO:teuthology.orchestra.run.smithi112.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T13:44:46.663 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: amd64 2024-09-13T13:44:46.663 INFO:teuthology.orchestra.run.smithi112.stdout:Source: linux-meta 2024-09-13T13:44:46.663 INFO:teuthology.orchestra.run.smithi112.stdout:Version: 5.15.0.121.121 2024-09-13T13:44:46.663 INFO:teuthology.orchestra.run.smithi112.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-13T13:44:46.663 INFO:teuthology.orchestra.run.smithi112.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T13:44:46.663 INFO:teuthology.orchestra.run.smithi112.stdout:Recommends: thermald 2024-09-13T13:44:46.663 INFO:teuthology.orchestra.run.smithi112.stdout:Description: Generic Linux kernel image 2024-09-13T13:44:46.663 INFO:teuthology.orchestra.run.smithi112.stdout: This package will always depend on the latest generic kernel image 2024-09-13T13:44:46.664 INFO:teuthology.orchestra.run.smithi112.stdout: available. 2024-09-13T13:44:46.664 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2024-09-13T13:44:46.708 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.6lvJfDumhI 2024-09-13T13:44:46.708 DEBUG:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub/grub.cfg /tmp/tmp.6lvJfDumhI 2024-09-13T13:44:46.763 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.6lvJfDumhI 2024-09-13T13:44:46.871 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.6lvJfDumhI is 10KB 2024-09-13T13:44:46.883 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.6lvJfDumhI 2024-09-13T13:44:46.888 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-13T13:44:46.943 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-13T13:44:46.943 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-13T13:44:46.943 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-13T13:44:47.010 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-13T13:44:47.010 DEBUG:teuthology.orchestra.run.smithi112:> sudo update-grub 2024-09-13T13:44:48.198 INFO:teuthology.orchestra.run.smithi112.stderr:Sourcing file `/etc/default/grub' 2024-09-13T13:44:48.200 INFO:teuthology.orchestra.run.smithi112.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T13:44:48.202 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2024-09-13T13:44:48.386 INFO:teuthology.orchestra.run.smithi112.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-13T13:44:48.392 INFO:teuthology.orchestra.run.smithi112.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-13T13:44:48.616 INFO:teuthology.orchestra.run.smithi112.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-13T13:44:48.621 INFO:teuthology.orchestra.run.smithi112.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-13T13:44:48.663 INFO:teuthology.orchestra.run.smithi112.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-13T13:44:48.668 INFO:teuthology.orchestra.run.smithi112.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-13T13:44:48.775 INFO:teuthology.orchestra.run.smithi112.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-13T13:44:48.775 INFO:teuthology.orchestra.run.smithi112.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-13T13:44:48.775 INFO:teuthology.orchestra.run.smithi112.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-13T13:44:48.797 INFO:teuthology.orchestra.run.smithi112.stderr:done 2024-09-13T13:44:48.813 DEBUG:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2024-09-13T13:45:18.845 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T13:45:18.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-13T13:45:18.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:45:53.473 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-13T13:46:02.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-13T13:46:02.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:46:20.861 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-13T13:46:32.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-13T13:46:32.874 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:46:32.884 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2024-09-13T13:46:47.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-13T13:46:47.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:46:48.166 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-09-13T13:46:48.708 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-13T13:46:48.708 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-13T13:46:48.708 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2024-09-13T13:46:48.757 INFO:teuthology.orchestra.run.smithi112.stdout:5.15.0-121-generic 2024-09-13T13:46:48.757 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-13T13:46:48.757 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T13:46:48.757 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-13T13:46:49.758 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T13:46:49.758 DEBUG:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T13:46:50.116 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2024-09-13T13:46:50.135 DEBUG:teuthology.parallel:result is None 2024-09-13T13:46:50.135 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T13:46:50.143 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T13:46:50.143 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T13:46:50.148 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T13:46:50.185 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T13:46:50.235 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T13:46:50.235 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T13:46:50.257 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T13:46:50.285 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T13:46:50.285 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T13:46:50.314 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:46:50.322 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:46:50.324 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T13:46:50.331 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T13:46:50.331 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T13:46:50.380 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T13:46:50.389 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T13:46:50.389 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T13:46:50.429 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T13:46:50.477 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T13:46:50.525 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-13T13:46:50.525 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T13:46:50.593 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2024-09-13T13:46:50.666 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T13:46:50.673 INFO:teuthology.task.internal:Starting timer... 2024-09-13T13:46:50.673 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T13:46:50.722 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T13:46:50.763 DEBUG:teuthology.task.selinux:Excluding smithi112: OS 'ubuntu' does not support SELinux 2024-09-13T13:46:50.763 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T13:46:50.763 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-13T13:46:50.763 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T13:46:50.763 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T13:46:50.796 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-13T13:46:50.800 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-13T13:46:50.961 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T13:46:50.972 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T13:46:50.973 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi112.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T13:54:15.001 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi112.front.sepia.ceph.com')] 2024-09-13T13:54:15.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-13T13:54:15.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T13:54:15.106 DEBUG:teuthology.orchestra.run.smithi112:> true 2024-09-13T13:54:15.190 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2024-09-13T13:54:15.191 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T13:54:15.204 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T13:54:15.204 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T13:54:15.204 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T13:54:15.260 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-13T13:54:15.261 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: Command line: ntpd -gq 2024-09-13T13:54:15.261 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: ---------------------------------------------------- 2024-09-13T13:54:15.261 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: ntp-4 is maintained by Network Time Foundation, 2024-09-13T13:54:15.261 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-13T13:54:15.261 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: corporation. Support and training for ntp-4 are 2024-09-13T13:54:15.261 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: available at https://www.nwtime.org/support 2024-09-13T13:54:15.261 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: ---------------------------------------------------- 2024-09-13T13:54:15.262 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: proto: precision = 0.043 usec (-24) 2024-09-13T13:54:15.262 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: basedate set to 2022-02-04 2024-09-13T13:54:15.262 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: gps base set to 2022-02-06 (week 2196) 2024-09-13T13:54:15.262 INFO:teuthology.orchestra.run.smithi112.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T13:54:15.262 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T13:54:15.263 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: restrict ::: KOD does nothing without LIMITED. 2024-09-13T13:54:15.263 INFO:teuthology.orchestra.run.smithi112.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-13T13:54:15.264 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: Listen and drop on 0 v6wildcard [::]:123 2024-09-13T13:54:15.264 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-13T13:54:15.264 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: Listen normally on 2 lo 127.0.0.1:123 2024-09-13T13:54:15.264 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: Listen normally on 3 enp3s0f1 172.21.15.112:123 2024-09-13T13:54:15.264 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: Listen normally on 4 lo [::1]:123 2024-09-13T13:54:15.264 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cb69%5]:123 2024-09-13T13:54:15.264 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:15 ntpd[18486]: Listening on routing socket on fd #22 for interface updates 2024-09-13T13:54:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:17 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:17 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:17 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:17 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:17 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T13:54:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:17 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T13:54:18.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:18 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:18.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:18 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:19.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:19 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:19.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:19 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:19.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:19 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:19.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:19 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:19.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:19 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T13:54:19.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:19 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T13:54:20.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:20 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:20.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:20 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:20.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:20 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:20.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:20 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:20.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:20 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:20.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:20 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:20.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:20 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T13:54:20.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:20 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T13:54:20.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:20 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:20.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:20 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:20.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:20 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T13:54:20.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:20 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T13:54:21.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:21 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:21.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:21 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:21.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:21 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:21.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:21 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:21.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:21 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T13:54:21.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:21 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T13:54:22.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:22 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:22.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:22 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:22.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:22 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:22.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:22 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:22.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:22 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:22.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:22 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:22.263 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:22 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T13:54:22.263 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:22 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T13:54:22.263 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:22 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:22.263 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:22 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:22.263 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:22 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T13:54:22.263 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:22 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T13:54:23.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:23 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:23.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:23 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:23.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:23 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T13:54:23.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:23 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T13:54:23.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:23 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T13:54:23.261 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:23 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T13:54:23.262 INFO:teuthology.orchestra.run.smithi112.stdout:13 Sep 13:54:23 ntpd[18486]: ntpd: time slew -0.002049 s 2024-09-13T13:54:23.262 INFO:teuthology.orchestra.run.smithi112.stdout:ntpd: time slew -0.002049s 2024-09-13T13:54:23.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:23 ntpd[18486]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-13T13:54:23.262 INFO:teuthology.orchestra.run.smithi112.stderr:13 Sep 13:54:23 ntpd[18486]: can't open /var/log/ntpstats/loopstats.20240913: Permission denied 2024-09-13T13:54:23.317 INFO:teuthology.orchestra.run.smithi112.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T13:54:23.317 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================== 2024-09-13T13:54:23.317 INFO:teuthology.orchestra.run.smithi112.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T13:54:23.317 INFO:teuthology.orchestra.run.smithi112.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T13:54:23.317 INFO:teuthology.orchestra.run.smithi112.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T13:54:23.317 INFO:teuthology.orchestra.run.smithi112.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-13T13:54:23.318 INFO:teuthology.run_tasks:Running task install... 2024-09-13T13:54:23.328 DEBUG:teuthology.task.install:project ceph 2024-09-13T13:54:23.328 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'}} 2024-09-13T13:54:23.328 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd58f73069a8157a9ace12a0247ee3e8497707ea9'} 2024-09-13T13:54:23.328 INFO:teuthology.task.install:Using flavor: default 2024-09-13T13:54:23.335 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T13:54:23.335 INFO:teuthology.task.install:extra packages: [] 2024-09-13T13:54:23.336 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-key list | grep Ceph 2024-09-13T13:54:23.417 INFO:teuthology.orchestra.run.smithi112.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-13T13:54:23.447 INFO:teuthology.orchestra.run.smithi112.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-13T13:54:23.447 INFO:teuthology.orchestra.run.smithi112.stdout:uid [ unknown] Ceph.com (release key) 2024-09-13T13:54:23.448 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-13T13:54:23.448 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:54:23.595 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default/ 2024-09-13T13:54:23.595 INFO:teuthology.task.install.deb:Package version is 19.3.0-4697-gd58f7306-1jammy 2024-09-13T13:54:23.714 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-13T13:54:23.714 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-13T13:54:23.727 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get update 2024-09-13T13:54:23.953 INFO:teuthology.orchestra.run.smithi112.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy InRelease 2024-09-13T13:54:23.974 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-13T13:54:23.998 INFO:teuthology.orchestra.run.smithi112.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-13T13:54:24.019 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,426 B] 2024-09-13T13:54:24.037 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-13T13:54:24.037 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-13T13:54:24.059 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-13T13:54:24.101 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-13T13:54:24.183 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-13T13:54:24.412 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 29.3 kB in 0s (58.9 kB/s) 2024-09-13T13:54:25.426 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-13T13:54:25.443 DEBUG:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4697-gd58f7306-1jammy cephadm=19.3.0-4697-gd58f7306-1jammy ceph-mds=19.3.0-4697-gd58f7306-1jammy ceph-mgr=19.3.0-4697-gd58f7306-1jammy ceph-common=19.3.0-4697-gd58f7306-1jammy ceph-fuse=19.3.0-4697-gd58f7306-1jammy ceph-test=19.3.0-4697-gd58f7306-1jammy ceph-volume=19.3.0-4697-gd58f7306-1jammy radosgw=19.3.0-4697-gd58f7306-1jammy python3-rados=19.3.0-4697-gd58f7306-1jammy python3-rgw=19.3.0-4697-gd58f7306-1jammy python3-cephfs=19.3.0-4697-gd58f7306-1jammy python3-rbd=19.3.0-4697-gd58f7306-1jammy libcephfs2=19.3.0-4697-gd58f7306-1jammy libcephfs-dev=19.3.0-4697-gd58f7306-1jammy librados2=19.3.0-4697-gd58f7306-1jammy librbd1=19.3.0-4697-gd58f7306-1jammy rbd-fuse=19.3.0-4697-gd58f7306-1jammy 2024-09-13T13:54:25.498 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2024-09-13T13:54:25.683 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2024-09-13T13:54:25.684 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2024-09-13T13:54:25.866 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T13:54:25.866 INFO:teuthology.orchestra.run.smithi112.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-13T13:54:25.867 INFO:teuthology.orchestra.run.smithi112.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-13T13:54:25.867 INFO:teuthology.orchestra.run.smithi112.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-13T13:54:25.867 INFO:teuthology.orchestra.run.smithi112.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-13T13:54:25.867 INFO:teuthology.orchestra.run.smithi112.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-13T13:54:25.867 INFO:teuthology.orchestra.run.smithi112.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T13:54:25.868 INFO:teuthology.orchestra.run.smithi112.stdout:The following additional packages will be installed: 2024-09-13T13:54:25.868 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-13T13:54:25.868 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-13T13:54:25.868 INFO:teuthology.orchestra.run.smithi112.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-13T13:54:25.868 INFO:teuthology.orchestra.run.smithi112.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-13T13:54:25.869 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-13T13:54:25.869 INFO:teuthology.orchestra.run.smithi112.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-13T13:54:25.870 INFO:teuthology.orchestra.run.smithi112.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-13T13:54:25.870 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-13T13:54:25.870 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-13T13:54:25.870 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T13:54:25.870 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T13:54:25.870 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T13:54:25.870 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T13:54:25.870 INFO:teuthology.orchestra.run.smithi112.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T13:54:25.870 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T13:54:25.870 INFO:teuthology.orchestra.run.smithi112.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T13:54:25.870 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-13T13:54:25.870 INFO:teuthology.orchestra.run.smithi112.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-13T13:54:25.870 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-13T13:54:25.871 INFO:teuthology.orchestra.run.smithi112.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-13T13:54:25.871 INFO:teuthology.orchestra.run.smithi112.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-13T13:54:25.871 INFO:teuthology.orchestra.run.smithi112.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-13T13:54:25.871 INFO:teuthology.orchestra.run.smithi112.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-13T13:54:25.871 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet zip 2024-09-13T13:54:25.872 INFO:teuthology.orchestra.run.smithi112.stdout:Suggested packages: 2024-09-13T13:54:25.872 INFO:teuthology.orchestra.run.smithi112.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-13T13:54:25.872 INFO:teuthology.orchestra.run.smithi112.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-13T13:54:25.872 INFO:teuthology.orchestra.run.smithi112.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-13T13:54:25.872 INFO:teuthology.orchestra.run.smithi112.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-13T13:54:25.872 INFO:teuthology.orchestra.run.smithi112.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-13T13:54:25.872 INFO:teuthology.orchestra.run.smithi112.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-13T13:54:25.872 INFO:teuthology.orchestra.run.smithi112.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-13T13:54:25.872 INFO:teuthology.orchestra.run.smithi112.stdout:Recommended packages: 2024-09-13T13:54:25.872 INFO:teuthology.orchestra.run.smithi112.stdout: btrfs-tools 2024-09-13T13:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout:The following NEW packages will be installed: 2024-09-13T13:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-13T13:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-13T13:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-13T13:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-13T13:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-13T13:54:25.975 INFO:teuthology.orchestra.run.smithi112.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-13T13:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-13T13:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-13T13:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-13T13:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-13T13:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-13T13:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-13T13:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-13T13:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-13T13:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-13T13:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-13T13:54:25.977 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-13T13:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-13T13:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-13T13:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-13T13:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-13T13:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-13T13:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-13T13:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-13T13:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-13T13:54:25.978 INFO:teuthology.orchestra.run.smithi112.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-13T13:54:26.122 INFO:teuthology.orchestra.run.smithi112.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-13T13:54:26.122 INFO:teuthology.orchestra.run.smithi112.stdout:Need to get 251 MB of archives. 2024-09-13T13:54:26.122 INFO:teuthology.orchestra.run.smithi112.stdout:After this operation, 933 MB of additional disk space will be used. 2024-09-13T13:54:26.122 INFO:teuthology.orchestra.run.smithi112.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4697-gd58f7306-1jammy [3,631 kB] 2024-09-13T13:54:26.166 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-13T13:54:26.342 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4697-gd58f7306-1jammy [3,320 kB] 2024-09-13T13:54:26.416 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4697-gd58f7306-1jammy [722 kB] 2024-09-13T13:54:26.431 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4697-gd58f7306-1jammy [356 kB] 2024-09-13T13:54:26.437 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4697-gd58f7306-1jammy [32.8 kB] 2024-09-13T13:54:26.438 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4697-gd58f7306-1jammy [180 kB] 2024-09-13T13:54:26.441 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4697-gd58f7306-1jammy [71.2 kB] 2024-09-13T13:54:26.443 INFO:teuthology.orchestra.run.smithi112.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4697-gd58f7306-1jammy [338 kB] 2024-09-13T13:54:26.451 INFO:teuthology.orchestra.run.smithi112.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4697-gd58f7306-1jammy [7,979 kB] 2024-09-13T13:54:26.584 INFO:teuthology.orchestra.run.smithi112.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4697-gd58f7306-1jammy [112 kB] 2024-09-13T13:54:26.586 INFO:teuthology.orchestra.run.smithi112.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4697-gd58f7306-1jammy [473 kB] 2024-09-13T13:54:26.587 INFO:teuthology.orchestra.run.smithi112.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-13T13:54:26.632 INFO:teuthology.orchestra.run.smithi112.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4697-gd58f7306-1jammy [28.2 MB] 2024-09-13T13:54:26.632 INFO:teuthology.orchestra.run.smithi112.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-13T13:54:26.681 INFO:teuthology.orchestra.run.smithi112.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-13T13:54:26.909 INFO:teuthology.orchestra.run.smithi112.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-13T13:54:26.920 INFO:teuthology.orchestra.run.smithi112.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-13T13:54:26.949 INFO:teuthology.orchestra.run.smithi112.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-13T13:54:26.958 INFO:teuthology.orchestra.run.smithi112.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-13T13:54:26.959 INFO:teuthology.orchestra.run.smithi112.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-13T13:54:26.960 INFO:teuthology.orchestra.run.smithi112.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-13T13:54:26.979 INFO:teuthology.orchestra.run.smithi112.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-13T13:54:26.985 INFO:teuthology.orchestra.run.smithi112.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-13T13:54:26.989 INFO:teuthology.orchestra.run.smithi112.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-13T13:54:26.992 INFO:teuthology.orchestra.run.smithi112.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-13T13:54:27.074 INFO:teuthology.orchestra.run.smithi112.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-13T13:54:27.137 INFO:teuthology.orchestra.run.smithi112.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-13T13:54:27.189 INFO:teuthology.orchestra.run.smithi112.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-13T13:54:27.190 INFO:teuthology.orchestra.run.smithi112.stdout:Get:30 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4697-gd58f7306-1jammy [5,206 kB] 2024-09-13T13:54:27.193 INFO:teuthology.orchestra.run.smithi112.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-13T13:54:27.194 INFO:teuthology.orchestra.run.smithi112.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-13T13:54:27.195 INFO:teuthology.orchestra.run.smithi112.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-13T13:54:27.197 INFO:teuthology.orchestra.run.smithi112.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-13T13:54:27.221 INFO:teuthology.orchestra.run.smithi112.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-13T13:54:27.221 INFO:teuthology.orchestra.run.smithi112.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-13T13:54:27.222 INFO:teuthology.orchestra.run.smithi112.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-13T13:54:27.237 INFO:teuthology.orchestra.run.smithi112.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-13T13:54:27.255 INFO:teuthology.orchestra.run.smithi112.stdout:Get:39 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4697-gd58f7306-1jammy [251 kB] 2024-09-13T13:54:27.258 INFO:teuthology.orchestra.run.smithi112.stdout:Get:40 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4697-gd58f7306-1jammy [124 kB] 2024-09-13T13:54:27.260 INFO:teuthology.orchestra.run.smithi112.stdout:Get:41 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4697-gd58f7306-1jammy [1,561 kB] 2024-09-13T13:54:27.273 INFO:teuthology.orchestra.run.smithi112.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4697-gd58f7306-1jammy [6,309 kB] 2024-09-13T13:54:27.305 INFO:teuthology.orchestra.run.smithi112.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-13T13:54:27.306 INFO:teuthology.orchestra.run.smithi112.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-13T13:54:27.311 INFO:teuthology.orchestra.run.smithi112.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-13T13:54:27.312 INFO:teuthology.orchestra.run.smithi112.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-13T13:54:27.313 INFO:teuthology.orchestra.run.smithi112.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-13T13:54:27.314 INFO:teuthology.orchestra.run.smithi112.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-13T13:54:27.316 INFO:teuthology.orchestra.run.smithi112.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-13T13:54:27.316 INFO:teuthology.orchestra.run.smithi112.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-13T13:54:27.318 INFO:teuthology.orchestra.run.smithi112.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-13T13:54:27.365 INFO:teuthology.orchestra.run.smithi112.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4697-gd58f7306-1jammy [23.1 MB] 2024-09-13T13:54:27.399 INFO:teuthology.orchestra.run.smithi112.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-13T13:54:27.400 INFO:teuthology.orchestra.run.smithi112.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-13T13:54:27.401 INFO:teuthology.orchestra.run.smithi112.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-13T13:54:27.412 INFO:teuthology.orchestra.run.smithi112.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-13T13:54:27.412 INFO:teuthology.orchestra.run.smithi112.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-13T13:54:27.413 INFO:teuthology.orchestra.run.smithi112.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-13T13:54:27.414 INFO:teuthology.orchestra.run.smithi112.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-13T13:54:27.417 INFO:teuthology.orchestra.run.smithi112.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-13T13:54:27.421 INFO:teuthology.orchestra.run.smithi112.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-13T13:54:27.482 INFO:teuthology.orchestra.run.smithi112.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-13T13:54:27.504 INFO:teuthology.orchestra.run.smithi112.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-13T13:54:27.505 INFO:teuthology.orchestra.run.smithi112.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-13T13:54:27.510 INFO:teuthology.orchestra.run.smithi112.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-13T13:54:27.511 INFO:teuthology.orchestra.run.smithi112.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-13T13:54:27.512 INFO:teuthology.orchestra.run.smithi112.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-13T13:54:27.567 INFO:teuthology.orchestra.run.smithi112.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-13T13:54:27.730 INFO:teuthology.orchestra.run.smithi112.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-13T13:54:27.731 INFO:teuthology.orchestra.run.smithi112.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-13T13:54:27.733 INFO:teuthology.orchestra.run.smithi112.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-13T13:54:27.734 INFO:teuthology.orchestra.run.smithi112.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-13T13:54:27.735 INFO:teuthology.orchestra.run.smithi112.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-13T13:54:27.813 INFO:teuthology.orchestra.run.smithi112.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-13T13:54:27.891 INFO:teuthology.orchestra.run.smithi112.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-13T13:54:27.892 INFO:teuthology.orchestra.run.smithi112.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4697-gd58f7306-1jammy [14.1 kB] 2024-09-13T13:54:27.892 INFO:teuthology.orchestra.run.smithi112.stdout:Get:77 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4697-gd58f7306-1jammy [880 kB] 2024-09-13T13:54:27.892 INFO:teuthology.orchestra.run.smithi112.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-13T13:54:27.904 INFO:teuthology.orchestra.run.smithi112.stdout:Get:79 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4697-gd58f7306-1jammy [2,433 kB] 2024-09-13T13:54:27.923 INFO:teuthology.orchestra.run.smithi112.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-13T13:54:27.923 INFO:teuthology.orchestra.run.smithi112.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-13T13:54:27.925 INFO:teuthology.orchestra.run.smithi112.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-13T13:54:27.925 INFO:teuthology.orchestra.run.smithi112.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-13T13:54:27.926 INFO:teuthology.orchestra.run.smithi112.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-13T13:54:27.927 INFO:teuthology.orchestra.run.smithi112.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-13T13:54:27.937 INFO:teuthology.orchestra.run.smithi112.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4697-gd58f7306-1jammy [812 kB] 2024-09-13T13:54:27.947 INFO:teuthology.orchestra.run.smithi112.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4697-gd58f7306-1jammy [160 kB] 2024-09-13T13:54:27.949 INFO:teuthology.orchestra.run.smithi112.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4697-gd58f7306-1jammy [61.4 MB] 2024-09-13T13:54:27.978 INFO:teuthology.orchestra.run.smithi112.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-13T13:54:27.979 INFO:teuthology.orchestra.run.smithi112.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-13T13:54:28.059 INFO:teuthology.orchestra.run.smithi112.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-13T13:54:28.064 INFO:teuthology.orchestra.run.smithi112.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-13T13:54:28.067 INFO:teuthology.orchestra.run.smithi112.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-13T13:54:28.068 INFO:teuthology.orchestra.run.smithi112.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-13T13:54:28.077 INFO:teuthology.orchestra.run.smithi112.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-13T13:54:28.083 INFO:teuthology.orchestra.run.smithi112.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-13T13:54:28.085 INFO:teuthology.orchestra.run.smithi112.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-13T13:54:28.086 INFO:teuthology.orchestra.run.smithi112.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-13T13:54:28.140 INFO:teuthology.orchestra.run.smithi112.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-13T13:54:28.142 INFO:teuthology.orchestra.run.smithi112.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-13T13:54:28.222 INFO:teuthology.orchestra.run.smithi112.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-13T13:54:28.223 INFO:teuthology.orchestra.run.smithi112.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-13T13:54:28.224 INFO:teuthology.orchestra.run.smithi112.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-13T13:54:28.225 INFO:teuthology.orchestra.run.smithi112.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-13T13:54:28.229 INFO:teuthology.orchestra.run.smithi112.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-13T13:54:28.231 INFO:teuthology.orchestra.run.smithi112.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-13T13:54:28.248 INFO:teuthology.orchestra.run.smithi112.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-13T13:54:28.248 INFO:teuthology.orchestra.run.smithi112.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-13T13:54:28.303 INFO:teuthology.orchestra.run.smithi112.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-13T13:54:28.308 INFO:teuthology.orchestra.run.smithi112.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-13T13:54:28.385 INFO:teuthology.orchestra.run.smithi112.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-13T13:54:29.173 INFO:teuthology.orchestra.run.smithi112.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4697-gd58f7306-1jammy [8,625 kB] 2024-09-13T13:54:29.279 INFO:teuthology.orchestra.run.smithi112.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4697-gd58f7306-1jammy [14.2 kB] 2024-09-13T13:54:29.280 INFO:teuthology.orchestra.run.smithi112.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4697-gd58f7306-1jammy [55.7 MB] 2024-09-13T13:54:30.432 INFO:teuthology.orchestra.run.smithi112.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4697-gd58f7306-1jammy [130 kB] 2024-09-13T13:54:30.434 INFO:teuthology.orchestra.run.smithi112.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4697-gd58f7306-1jammy [38.9 kB] 2024-09-13T13:54:30.497 INFO:teuthology.orchestra.run.smithi112.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4697-gd58f7306-1jammy [15.9 MB] 2024-09-13T13:54:30.799 INFO:teuthology.orchestra.run.smithi112.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-aclamk-cbt-combined/d58f73069a8157a9ace12a0247ee3e8497707ea9/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4697-gd58f7306-1jammy [91.3 kB] 2024-09-13T13:54:31.371 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 251 MB in 5s (52.1 MB/s) 2024-09-13T13:54:31.529 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-13T13:54:31.630 INFO:teuthology.orchestra.run.smithi112.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-13T13:54:31.635 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-13T13:54:31.720 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T13:54:32.073 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-13T13:54:32.100 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-13T13:54:32.140 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T13:54:32.426 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-13T13:54:32.441 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-13T13:54:32.476 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T13:54:32.921 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-13T13:54:32.948 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T13:54:32.997 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T13:54:33.391 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-13T13:54:33.418 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T13:54:33.458 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T13:54:33.753 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-13T13:54:33.779 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-13T13:54:33.819 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T13:54:34.164 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-13T13:54:34.190 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-13T13:54:34.230 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T13:54:34.550 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librados2. 2024-09-13T13:54:34.576 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../007-librados2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:54:34.616 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librados2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:35.061 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librbd1. 2024-09-13T13:54:35.087 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../008-librbd1_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:54:35.119 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librbd1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:35.564 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libcephfs2. 2024-09-13T13:54:35.590 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../009-libcephfs2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:54:35.631 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libcephfs2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:35.908 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rados. 2024-09-13T13:54:35.923 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../010-python3-rados_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:54:35.966 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rados (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:36.294 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-13T13:54:36.320 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T13:54:36.360 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-ceph-argparse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:36.588 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cephfs. 2024-09-13T13:54:36.614 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../012-python3-cephfs_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:54:36.646 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cephfs (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:36.974 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-13T13:54:37.001 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../013-python3-ceph-common_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T13:54:37.041 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:37.386 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-13T13:54:37.412 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-13T13:54:37.452 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T13:54:37.755 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-prettytable. 2024-09-13T13:54:37.782 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-13T13:54:37.821 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-13T13:54:38.066 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rbd. 2024-09-13T13:54:38.081 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../016-python3-rbd_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:54:38.116 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rbd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:38.677 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-13T13:54:38.704 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-13T13:54:38.744 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T13:54:39.021 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-13T13:54:39.036 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-13T13:54:39.079 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T13:54:39.374 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-13T13:54:39.390 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-13T13:54:39.432 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T13:54:39.761 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package lua5.1. 2024-09-13T13:54:39.787 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-13T13:54:39.827 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-13T13:54:40.139 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package lua-any. 2024-09-13T13:54:40.165 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-13T13:54:40.205 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-13T13:54:40.441 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package zip. 2024-09-13T13:54:40.464 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-13T13:54:40.499 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking zip (3.0-12build2) ... 2024-09-13T13:54:40.752 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package unzip. 2024-09-13T13:54:40.779 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-13T13:54:40.811 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-13T13:54:41.231 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package luarocks. 2024-09-13T13:54:41.258 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-13T13:54:41.298 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-13T13:54:41.680 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librgw2. 2024-09-13T13:54:41.707 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../025-librgw2_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:54:41.747 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librgw2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:42.349 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rgw. 2024-09-13T13:54:42.373 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../026-python3-rgw_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:54:42.408 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:42.727 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-13T13:54:42.754 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-13T13:54:42.794 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T13:54:43.097 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libradosstriper1. 2024-09-13T13:54:43.124 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../028-libradosstriper1_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:54:43.163 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libradosstriper1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:43.558 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-common. 2024-09-13T13:54:43.585 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../029-ceph-common_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:54:43.625 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:45.080 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-base. 2024-09-13T13:54:45.107 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../030-ceph-base_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:54:45.156 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-base (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:45.751 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-13T13:54:45.778 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-13T13:54:45.818 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-13T13:54:46.137 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cheroot. 2024-09-13T13:54:46.164 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-13T13:54:46.204 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T13:54:46.499 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-13T13:54:46.525 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-13T13:54:46.557 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-13T13:54:46.843 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-13T13:54:46.869 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-13T13:54:46.909 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-13T13:54:47.212 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-13T13:54:47.239 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-13T13:54:47.279 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-13T13:54:47.582 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-tempora. 2024-09-13T13:54:47.609 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-13T13:54:47.648 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-13T13:54:47.910 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-portend. 2024-09-13T13:54:47.936 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-13T13:54:47.968 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-13T13:54:48.262 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-13T13:54:48.289 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-13T13:54:48.329 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-13T13:54:48.666 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-13T13:54:48.692 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-13T13:54:48.732 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-13T13:54:49.243 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-natsort. 2024-09-13T13:54:49.270 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-13T13:54:49.310 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-13T13:54:49.622 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-logutils. 2024-09-13T13:54:49.648 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-13T13:54:49.688 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-13T13:54:50.000 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-13T13:54:50.026 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-13T13:54:50.066 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-13T13:54:50.319 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-mako. 2024-09-13T13:54:50.346 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-13T13:54:50.385 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T13:54:50.689 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-13T13:54:50.715 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-13T13:54:50.755 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-13T13:54:51.058 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-13T13:54:51.085 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-13T13:54:51.125 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-13T13:54:51.386 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-webob. 2024-09-13T13:54:51.412 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-13T13:54:51.452 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T13:54:51.764 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-waitress. 2024-09-13T13:54:51.790 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-13T13:54:51.825 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T13:54:52.166 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-tempita. 2024-09-13T13:54:52.193 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-13T13:54:52.233 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T13:54:52.511 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-paste. 2024-09-13T13:54:52.528 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-13T13:54:52.561 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T13:54:52.872 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-13T13:54:52.899 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-13T13:54:52.939 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T13:54:53.242 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-13T13:54:53.268 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-13T13:54:53.308 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-13T13:54:53.636 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-webtest. 2024-09-13T13:54:53.663 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-13T13:54:53.703 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-13T13:54:54.072 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pecan. 2024-09-13T13:54:54.099 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-13T13:54:54.181 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T13:54:54.544 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-certifi. 2024-09-13T13:54:54.571 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-13T13:54:54.611 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-13T13:54:54.872 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-idna. 2024-09-13T13:54:54.898 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-13T13:54:54.938 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T13:54:55.258 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-urllib3. 2024-09-13T13:54:55.285 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-13T13:54:55.325 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T13:54:55.636 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-requests. 2024-09-13T13:54:55.663 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-13T13:54:55.703 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T13:54:56.006 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-13T13:54:56.033 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-13T13:54:56.072 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T13:54:56.359 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-13T13:54:56.374 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T13:54:56.417 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:56.797 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-13T13:54:56.824 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:54:56.863 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:57.142 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-13T13:54:57.168 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-13T13:54:57.200 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-13T13:54:57.502 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-openssl. 2024-09-13T13:54:57.529 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-13T13:54:57.569 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-13T13:54:57.905 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr. 2024-09-13T13:54:57.932 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../063-ceph-mgr_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:54:57.972 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:58.384 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mon. 2024-09-13T13:54:58.410 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../064-ceph-mon_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:54:58.450 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mon (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:54:59.004 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-osd. 2024-09-13T13:54:59.030 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../065-ceph-osd_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:54:59.061 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-osd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:00.798 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph. 2024-09-13T13:55:00.813 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../066-ceph_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:55:00.856 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:01.176 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-fuse. 2024-09-13T13:55:01.202 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../067-ceph-fuse_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:55:01.242 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:01.588 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mds. 2024-09-13T13:55:01.614 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../068-ceph-mds_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:55:01.646 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mds (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:02.032 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package cephadm. 2024-09-13T13:55:02.059 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../069-cephadm_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:55:02.098 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:02.410 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-13T13:55:02.436 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-13T13:55:02.476 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-13T13:55:02.812 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-13T13:55:02.839 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T13:55:02.879 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T13:55:03.868 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-babel. 2024-09-13T13:55:03.894 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-13T13:55:03.934 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T13:55:04.262 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jinja2. 2024-09-13T13:55:04.288 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-13T13:55:04.328 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T13:55:04.673 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-13T13:55:04.700 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T13:55:04.739 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:05.076 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-13T13:55:05.103 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-13T13:55:05.143 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-13T13:55:05.429 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-routes. 2024-09-13T13:55:05.456 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-13T13:55:05.487 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T13:55:05.824 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-13T13:55:05.851 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T13:55:05.890 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:08.472 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-13T13:55:08.489 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-13T13:55:08.522 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T13:55:09.009 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-joblib. 2024-09-13T13:55:09.037 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-13T13:55:09.075 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T13:55:09.437 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-13T13:55:09.465 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-13T13:55:09.504 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-13T13:55:09.773 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-sklearn. 2024-09-13T13:55:09.789 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-13T13:55:09.823 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T13:55:10.482 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-13T13:55:10.510 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T13:55:10.634 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:11.771 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cachetools. 2024-09-13T13:55:11.799 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-13T13:55:11.837 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-13T13:55:12.140 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-13T13:55:12.167 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-13T13:55:12.223 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-13T13:55:12.560 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-13T13:55:12.588 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-13T13:55:12.627 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T13:55:12.938 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rsa. 2024-09-13T13:55:12.966 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-13T13:55:13.005 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-13T13:55:13.300 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-google-auth. 2024-09-13T13:55:13.328 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-13T13:55:13.366 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-13T13:55:13.670 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-13T13:55:13.698 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-13T13:55:13.736 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T13:55:14.048 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-websocket. 2024-09-13T13:55:14.076 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-13T13:55:14.114 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-13T13:55:14.426 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-13T13:55:14.454 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-13T13:55:14.492 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T13:55:15.173 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-13T13:55:15.190 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T13:55:15.231 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:15.492 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-13T13:55:15.520 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-13T13:55:15.550 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T13:55:15.887 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-13T13:55:15.915 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T13:55:15.953 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T13:55:16.214 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package jq. 2024-09-13T13:55:16.231 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-13T13:55:16.273 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-13T13:55:16.517 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package socat. 2024-09-13T13:55:16.533 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-13T13:55:16.575 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-13T13:55:16.854 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package xmlstarlet. 2024-09-13T13:55:16.870 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-13T13:55:16.912 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-13T13:55:17.291 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-test. 2024-09-13T13:55:17.304 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../097-ceph-test_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:55:17.341 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-test (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:19.846 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-volume. 2024-09-13T13:55:19.874 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../098-ceph-volume_19.3.0-4697-gd58f7306-1jammy_all.deb ... 2024-09-13T13:55:19.912 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-volume (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:20.190 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-13T13:55:20.207 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../099-libcephfs-dev_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:55:20.240 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libcephfs-dev (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:20.543 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-13T13:55:20.559 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-13T13:55:20.601 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T13:55:20.956 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-13T13:55:20.984 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-13T13:55:21.022 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-13T13:55:21.360 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package nvme-cli. 2024-09-13T13:55:21.388 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-13T13:55:21.426 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T13:55:21.798 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package pkg-config. 2024-09-13T13:55:21.827 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-13T13:55:21.856 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T13:55:22.101 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-13T13:55:22.117 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-13T13:55:22.159 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-13T13:55:22.596 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-13T13:55:22.624 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-13T13:55:22.662 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-13T13:55:22.965 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pastescript. 2024-09-13T13:55:22.993 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-13T13:55:23.031 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-13T13:55:23.352 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pluggy. 2024-09-13T13:55:23.380 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-13T13:55:23.418 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-13T13:55:23.713 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-psutil. 2024-09-13T13:55:23.741 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-13T13:55:23.779 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-13T13:55:24.124 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-py. 2024-09-13T13:55:24.152 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-13T13:55:24.224 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-13T13:55:24.552 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pygments. 2024-09-13T13:55:24.580 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-13T13:55:24.618 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T13:55:25.039 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-13T13:55:25.068 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-13T13:55:25.097 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-13T13:55:25.384 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-toml. 2024-09-13T13:55:25.412 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-13T13:55:25.450 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-13T13:55:25.761 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pytest. 2024-09-13T13:55:25.789 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-13T13:55:25.828 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T13:55:26.173 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-simplejson. 2024-09-13T13:55:26.201 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-13T13:55:26.239 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-13T13:55:26.508 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-13T13:55:26.525 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-13T13:55:26.567 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-13T13:55:27.271 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package radosgw. 2024-09-13T13:55:27.299 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../116-radosgw_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:55:27.337 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking radosgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:28.199 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package rbd-fuse. 2024-09-13T13:55:28.222 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../117-rbd-fuse_19.3.0-4697-gd58f7306-1jammy_amd64.deb ... 2024-09-13T13:55:28.265 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking rbd-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:28.553 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-13T13:55:28.820 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-13T13:55:28.946 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-13T13:55:29.210 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-13T13:55:29.513 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-13T13:55:29.926 INFO:teuthology.orchestra.run.smithi112.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-13T13:55:29.990 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:30.506 INFO:teuthology.orchestra.run.smithi112.stdout:Adding system user cephadm....done 2024-09-13T13:55:30.663 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-13T13:55:30.947 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-13T13:55:31.190 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-13T13:55:31.316 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-13T13:55:31.575 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-13T13:55:31.835 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-13T13:55:31.970 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-13T13:55:32.288 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-13T13:55:32.649 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-13T13:55:32.891 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-13T13:55:33.159 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-13T13:55:33.444 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-13T13:55:33.695 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-13T13:55:33.955 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-13T13:55:34.205 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-ceph-argparse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:34.440 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-13T13:55:34.549 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-13T13:55:34.650 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-13T13:55:34.776 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-13T13:55:35.035 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-13T13:55:35.553 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-13T13:55:35.584 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-13T13:55:35.647 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-13T13:55:35.765 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-13T13:55:36.116 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-13T13:55:36.225 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-13T13:55:36.493 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-13T13:55:36.761 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-13T13:55:37.062 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up zip (3.0-12build2) ... 2024-09-13T13:55:37.188 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-13T13:55:37.857 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-13T13:55:38.116 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-13T13:55:38.226 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-13T13:55:38.351 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-13T13:55:38.653 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-13T13:55:39.037 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-13T13:55:39.289 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-13T13:55:39.657 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-13T13:55:39.925 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-13T13:55:40.251 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-13T13:55:40.502 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-13T13:55:40.637 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:40.946 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-13T13:55:41.272 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-13T13:55:42.803 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-13T13:55:43.046 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T13:55:43.181 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-13T13:55:43.448 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-13T13:55:43.566 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-13T13:55:43.667 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-13T13:55:43.977 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-13T13:55:44.244 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-13T13:55:44.512 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T13:55:44.638 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-13T13:55:44.940 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-13T13:55:45.184 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-13T13:55:45.425 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-13T13:55:45.684 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-13T13:55:45.969 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-13T13:55:46.237 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-13T13:55:46.363 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-13T13:55:46.639 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-13T13:55:46.748 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-13T13:55:47.041 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-13T13:55:47.325 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-13T13:55:47.576 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-13T13:55:47.702 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-13T13:55:47.829 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-13T13:55:47.919 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-13T13:55:48.164 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-13T13:55:48.281 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-13T13:55:48.649 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-13T13:55:48.884 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-13T13:55:49.010 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-13T13:55:49.261 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-13T13:55:49.420 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-13T13:55:49.738 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-13T13:55:50.022 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-13T13:55:50.299 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-13T13:55:50.416 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-13T13:55:50.684 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-13T13:55:50.952 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-13T13:55:51.297 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-13T13:55:51.673 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-13T13:55:51.958 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-13T13:55:52.294 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-13T13:55:52.420 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librados2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:52.521 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:52.622 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-13T13:55:52.975 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libcephfs2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:53.101 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libradosstriper1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:53.227 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-13T13:55:53.514 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librbd1 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:53.640 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-modules-core (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:53.767 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:53.977 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T13:55:53.977 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T13:55:54.410 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libcephfs-dev (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:54.535 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rados (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:54.661 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librgw2 (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:54.787 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-13T13:55:55.991 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rbd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:56.117 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up rbd-fuse (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:56.243 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:56.369 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cephfs (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:56.512 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-common (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:57.047 INFO:teuthology.orchestra.run.smithi112.stdout:Adding group ceph....done 2024-09-13T13:55:57.365 INFO:teuthology.orchestra.run.smithi112.stdout:Adding system user ceph....done 2024-09-13T13:55:57.534 INFO:teuthology.orchestra.run.smithi112.stdout:Setting system user ceph properties....done 2024-09-13T13:55:57.550 INFO:teuthology.orchestra.run.smithi112.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-13T13:55:57.686 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-13T13:55:57.996 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-13T13:55:58.464 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-test (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:58.598 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up radosgw (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:59.081 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T13:55:59.081 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T13:55:59.551 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-base (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:55:59.906 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-13T13:56:00.361 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mds (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:56:00.591 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T13:56:00.591 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T13:56:01.030 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:56:01.260 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T13:56:01.260 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T13:56:01.918 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-osd (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:56:02.201 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T13:56:02.201 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T13:56:02.687 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:56:02.813 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:56:02.972 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mon (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:56:03.182 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T13:56:03.182 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T13:56:03.616 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-cephadm (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:56:03.800 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:56:03.925 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-dashboard (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:56:04.077 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-volume (19.3.0-4697-gd58f7306-1jammy) ... 2024-09-13T13:56:04.413 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-13T13:56:04.513 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-13T13:56:04.666 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-13T13:56:05.675 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-13T13:56:06.286 INFO:teuthology.orchestra.run.smithi112.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T13:56:06.291 DEBUG:teuthology.parallel:result is None 2024-09-13T13:56:06.291 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:56:06.444 DEBUG:teuthology.orchestra.run.smithi112:> dpkg-query -W -f '${Version}' ceph 2024-09-13T13:56:06.463 INFO:teuthology.orchestra.run.smithi112.stdout:19.3.0-4697-gd58f7306-1jammy 2024-09-13T13:56:06.463 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4697-gd58f7306-1jammy 2024-09-13T13:56:06.463 INFO:teuthology.task.install:The correct ceph version 19.3.0-4697-gd58f7306-1jammy is installed. 2024-09-13T13:56:06.465 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T13:56:06.465 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-13T13:56:06.465 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T13:56:06.519 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T13:56:06.519 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-13T13:56:06.519 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T13:56:06.575 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T13:56:06.631 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T13:56:06.631 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-13T13:56:06.631 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T13:56:06.687 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T13:56:06.742 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T13:56:06.742 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-13T13:56:06.742 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T13:56:06.794 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T13:56:06.847 INFO:teuthology.run_tasks:Running task workunit... 2024-09-13T13:56:06.860 INFO:tasks.workunit:Pulling workunits from ref d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:56:06.860 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-13T13:56:06.860 INFO:tasks.workunit:timeout=3h 2024-09-13T13:56:06.860 INFO:tasks.workunit:cleanup=True 2024-09-13T13:56:06.861 DEBUG:teuthology.orchestra.run.smithi112:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-13T13:56:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T13:56:06.896 INFO:teuthology.orchestra.run.smithi112.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-13T13:56:06.896 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T13:56:06.943 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-13T13:56:06.943 DEBUG:teuthology.orchestra.run.smithi112:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-13T13:56:06.991 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d58f73069a8157a9ace12a0247ee3e8497707ea9 2024-09-13T13:56:07.041 INFO:tasks.workunit.client.0.smithi112.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-13T13:57:03.743 INFO:tasks.workunit.client.0.smithi112.stderr:Note: switching to 'd58f73069a8157a9ace12a0247ee3e8497707ea9'. 2024-09-13T13:57:03.744 INFO:tasks.workunit.client.0.smithi112.stderr: 2024-09-13T13:57:03.744 INFO:tasks.workunit.client.0.smithi112.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-13T13:57:03.744 INFO:tasks.workunit.client.0.smithi112.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-13T13:57:03.744 INFO:tasks.workunit.client.0.smithi112.stderr:state without impacting any branches by switching back to a branch. 2024-09-13T13:57:03.744 INFO:tasks.workunit.client.0.smithi112.stderr: 2024-09-13T13:57:03.744 INFO:tasks.workunit.client.0.smithi112.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-13T13:57:03.744 INFO:tasks.workunit.client.0.smithi112.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-13T13:57:03.744 INFO:tasks.workunit.client.0.smithi112.stderr: 2024-09-13T13:57:03.744 INFO:tasks.workunit.client.0.smithi112.stderr: git switch -c 2024-09-13T13:57:03.745 INFO:tasks.workunit.client.0.smithi112.stderr: 2024-09-13T13:57:03.745 INFO:tasks.workunit.client.0.smithi112.stderr:Or undo this operation with: 2024-09-13T13:57:03.745 INFO:tasks.workunit.client.0.smithi112.stderr: 2024-09-13T13:57:03.745 INFO:tasks.workunit.client.0.smithi112.stderr: git switch - 2024-09-13T13:57:03.745 INFO:tasks.workunit.client.0.smithi112.stderr: 2024-09-13T13:57:03.745 INFO:tasks.workunit.client.0.smithi112.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-13T13:57:03.745 INFO:tasks.workunit.client.0.smithi112.stderr: 2024-09-13T13:57:03.745 INFO:tasks.workunit.client.0.smithi112.stderr:HEAD is now at d58f73069a8 tools/ceph-bluestore-tool: Added zap-device documentation 2024-09-13T13:57:03.749 DEBUG:teuthology.orchestra.run.smithi112:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-13T13:57:03.795 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2024-09-13T13:57:03.796 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-13T13:57:03.842 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-09-13T13:57:03.843 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-09-13T13:57:03.843 DEBUG:teuthology.orchestra.run.smithi112:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d58f73069a8157a9ace12a0247ee3e8497707ea9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-09-13T13:57:03.906 INFO:tasks.workunit.client.0.smithi112.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-13T13:57:03.912 INFO:tasks.workunit.client.0.smithi112.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-13T13:57:03.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-13T13:57:03.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-13T13:57:03.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-13T13:57:03.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-13T13:57:03.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-13T13:57:03.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-13T13:57:03.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-13T13:57:03.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-13T13:57:03.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-mapper 2024-09-13T13:57:03.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-09-13T13:57:03.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-09-13T13:57:03.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-09-13T13:57:03.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-09-13T13:57:03.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-09-13T13:57:03.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-09-13T13:57:03.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=e1ae251f-408a-4001-9aa9-ee4043234ba2 --auth-supported=none ' 2024-09-13T13:57:03.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-09-13T13:57:03.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-13T13:57:03.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-09-13T13:57:03.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-13T13:57:03.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-09-13T13:57:03.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-09-13T13:57:03.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-09-13T13:57:03.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-mapper 2024-09-13T13:57:03.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-mapper 2024-09-13T13:57:03.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2024-09-13T13:57:03.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T13:57:03.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2024-09-13T13:57:03.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T13:57:03.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T13:57:03.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T13:57:03.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T13:57:03.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T13:57:03.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T13:57:03.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T13:57:03.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T13:57:03.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T13:57:03.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T13:57:03.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T13:57:03.920 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T13:57:03.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:57:03.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T13:57:03.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T13:57:03.922 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T13:57:03.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T13:57:03.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T13:57:03.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2024-09-13T13:57:03.924 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T13:57:03.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T13:57:03.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21519 2024-09-13T13:57:03.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21519 2024-09-13T13:57:03.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T13:57:03.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T13:57:03.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-mapper 2024-09-13T13:57:03.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T13:57:03.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T13:57:03.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21519 2024-09-13T13:57:03.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21519 2024-09-13T13:57:03.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T13:57:03.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-13T13:57:03.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-13T13:57:03.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T13:57:03.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-09-13T13:57:03.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-09-13T13:57:03.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-09-13T13:57:03.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-09-13T13:57:03.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-09-13T13:57:03.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-09-13T13:57:03.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-09-13T13:57:03.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-09-13T13:57:03.930 INFO:tasks.workunit.client.0.smithi112.stdout:Dir: td/osd-mapper 2024-09-13T13:57:03.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-09-13T13:57:03.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-mapper 2024-09-13T13:57:03.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2024-09-13T13:57:03.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2024-09-13T13:57:03.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2024-09-13T13:57:03.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:234: standard_scrub_cluster: local poolname=test 2024-09-13T13:57:03.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: args['pool_name']=test 2024-09-13T13:57:03.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: local extra_pars= 2024-09-13T13:57:03.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:237: standard_scrub_cluster: local debug_msg=dbg 2024-09-13T13:57:03.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:240: standard_scrub_cluster: local saved_echo_flag=x 2024-09-13T13:57:03.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: set +x 2024-09-13T13:57:05.916 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 0b3f1c54-7083-45c9-9be6-5d59b7dc95d3 2024-09-13T13:57:06.155 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T13:57:06.277 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:06.272+0000 7f572c0237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:57:06.291 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:06.288+0000 7f572c0237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:57:06.292 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:06.288+0000 7f572c0237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:57:06.293 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:06.288+0000 7f572c0237c0 -1 bdev(0x55b30012bc00 td/osd-mapper/0/block) open stat got: (1) Operation not permitted 2024-09-13T13:57:06.293 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:06.288+0000 7f572c0237c0 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-09-13T13:57:12.421 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T13:57:12.621 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T13:57:12.651 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:12.648+0000 7f79768ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:57:12.654 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:12.652+0000 7f79768ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:57:12.656 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:12.652+0000 7f79768ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:57:12.837 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T13:57:13.960 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:13.956+0000 7f79768ee7c0 -1 Falling back to public interface 2024-09-13T13:57:14.047 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T13:57:15.249 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T13:57:16.570 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T13:57:16.887 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:16.884+0000 7f79768ee7c0 -1 osd.0 0 log_to_monitors true 2024-09-13T13:57:17.902 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T13:57:19.232 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T13:57:20.678 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T13:57:22.008 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T13:57:23.335 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T13:57:24.656 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T13:57:25.990 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T13:57:27.325 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T13:57:28.660 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T13:57:30.196 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T13:57:31.767 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T13:57:33.178 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T13:57:34.811 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T13:57:36.890 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T13:57:38.213 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T13:57:39.977 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T13:57:41.678 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T13:57:42.002 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/623105641,v1:127.0.0.1:6801/623105641] [v2:127.0.0.1:6802/623105641,v1:127.0.0.1:6803/623105641] exists,up 0b3f1c54-7083-45c9-9be6-5d59b7dc95d3 2024-09-13T13:57:42.007 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 d4e5e593-574b-4dbd-a048-ee4e8d283396 2024-09-13T13:57:42.462 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T13:57:42.519 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:42.520+0000 7f5c176077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:57:42.549 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:42.548+0000 7f5c176077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:57:42.549 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:42.548+0000 7f5c176077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:57:42.549 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:42.552+0000 7f5c176077c0 -1 bdev(0x562ad6264800 td/osd-mapper/1/block) open stat got: (1) Operation not permitted 2024-09-13T13:57:42.549 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:42.552+0000 7f5c176077c0 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-09-13T13:57:48.351 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T13:57:48.746 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T13:57:48.783 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:48.783+0000 7f26fd97f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:57:48.808 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:48.807+0000 7f26fd97f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:57:48.809 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:48.807+0000 7f26fd97f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:57:49.079 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T13:57:50.088 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:50.087+0000 7f26fd97f7c0 -1 Falling back to public interface 2024-09-13T13:57:50.381 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T13:57:51.706 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T13:57:52.798 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:57:52.799+0000 7f26fd97f7c0 -1 osd.1 0 log_to_monitors true 2024-09-13T13:57:53.045 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T13:57:54.480 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T13:57:55.906 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T13:57:57.238 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T13:57:58.569 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T13:57:59.941 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T13:58:01.279 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T13:58:02.624 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T13:58:03.965 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T13:58:05.302 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T13:58:07.050 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T13:58:08.683 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T13:58:10.280 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T13:58:11.879 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T13:58:13.248 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T13:58:15.113 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T13:58:15.437 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/830679600,v1:127.0.0.1:6811/830679600] [v2:127.0.0.1:6812/830679600,v1:127.0.0.1:6813/830679600] exists,up d4e5e593-574b-4dbd-a048-ee4e8d283396 2024-09-13T13:58:15.460 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 4657d173-9a8d-4d21-aeba-8aad33c6cb4a 2024-09-13T13:58:15.918 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T13:58:15.978 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:58:15.973+0000 7f6259cbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:58:16.044 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:58:16.041+0000 7f6259cbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:58:16.046 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:58:16.041+0000 7f6259cbd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:58:16.047 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:58:16.045+0000 7f6259cbd7c0 -1 bdev(0x5627388c7c00 td/osd-mapper/2/block) open stat got: (1) Operation not permitted 2024-09-13T13:58:16.047 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:58:16.045+0000 7f6259cbd7c0 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-09-13T13:58:22.189 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T13:58:22.594 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T13:58:22.632 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:58:22.629+0000 7f828d9367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:58:22.657 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:58:22.653+0000 7f828d9367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:58:22.659 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:58:22.657+0000 7f828d9367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:58:22.938 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T13:58:23.713 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:58:23.709+0000 7f828d9367c0 -1 Falling back to public interface 2024-09-13T13:58:24.278 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T13:58:25.596 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T13:58:26.480 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:58:26.477+0000 7f828d9367c0 -1 osd.2 0 log_to_monitors true 2024-09-13T13:58:26.916 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T13:58:28.474 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T13:58:29.803 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T13:58:31.124 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T13:58:32.454 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T13:58:33.789 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T13:58:35.115 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T13:58:36.451 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T13:58:37.785 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T13:58:39.117 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T13:58:40.433 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T13:58:42.200 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T13:58:43.621 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T13:58:45.295 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T13:58:46.856 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T13:58:48.745 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T13:58:50.144 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T13:58:50.467 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2494055309,v1:127.0.0.1:6819/2494055309] [v2:127.0.0.1:6820/2494055309,v1:127.0.0.1:6821/2494055309] exists,up 4657d173-9a8d-4d21-aeba-8aad33c6cb4a 2024-09-13T13:58:50.913 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'test' created 2024-09-13T13:58:53.131 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836501 2024-09-13T13:58:54.791 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672973 2024-09-13T13:58:55.143 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509447 2024-09-13T13:58:57.887 INFO:tasks.workunit.client.0.smithi112.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-09-13T13:58:57.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-09-13T13:58:58.026 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T13:58:58.027 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-09-13T13:58:58.027 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T13:58:58.039 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T13:58:58.040 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-09-13T13:58:58.040 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T13:58:58.052 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2: { 2024-09-13T13:58:58.052 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-09-13T13:58:58.052 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T13:58:58.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-09-13T13:58:58.193 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T13:58:58.194 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-09-13T13:58:58.194 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T13:58:58.207 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T13:58:58.207 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-09-13T13:58:58.207 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T13:58:58.220 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2: { 2024-09-13T13:58:58.220 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-09-13T13:58:58.220 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T13:58:58.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-09-13T13:58:58.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-09-13T13:58:58.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-09-13T13:58:58.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-09-13T13:58:58.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-09-13T13:58:58.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-09-13T13:58:58.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-09-13T13:58:58.547 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap01 2024-09-13T13:58:58.753 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap02 2024-09-13T13:58:58.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-09-13T13:58:58.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-09-13T13:58:58.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-09-13T13:58:59.201 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap13 2024-09-13T13:58:59.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-09-13T13:58:59.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-09-13T13:58:59.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-09-13T13:58:59.695 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap24 2024-09-13T13:58:59.900 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap25 2024-09-13T13:58:59.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 27212 2024-09-13T13:58:59.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-09-13T13:59:00.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-09-13T13:59:00.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-09-13T13:59:00.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-09-13T13:59:00.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-09-13T13:59:00.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-09-13T13:59:00.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-09-13T13:59:00.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-09-13T13:59:00.947 INFO:tasks.workunit.client.0.smithi112.stdout:pgid is 1.3 (primary: osd.1) 2024-09-13T13:59:00.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-09-13T13:59:00.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:303: set_query_debug: local pgid=1.3 2024-09-13T13:59:00.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:305: set_query_debug: ceph pg dump pgs_brief 2024-09-13T13:59:00.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:305: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-09-13T13:59:01.272 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs_brief 2024-09-13T13:59:01.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:305: set_query_debug: local prim_osd=1 2024-09-13T13:59:01.287 INFO:tasks.workunit.client.0.smithi112.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-09-13T13:59:01.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:307: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-09-13T13:59:01.287 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: set_query_debug: get_asok_path osd.1 2024-09-13T13:59:01.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T13:59:01.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T13:59:01.289 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T13:59:01.289 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T13:59:01.289 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21519 2024-09-13T13:59:01.289 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21519/ceph-osd.1.asok 2024-09-13T13:59:01.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: set_query_debug: CEPH_ARGS= 2024-09-13T13:59:01.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.21519/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-09-13T13:59:01.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-13T13:59:01.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-09-13T13:59:01.448 INFO:tasks.workunit.client.0.smithi112.stdout:{"success":true}{ 2024-09-13T13:59:01.448 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "objxxx", 2024-09-13T13:59:01.449 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T13:59:01.449 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:01.449 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 2, 2024-09-13T13:59:01.449 INFO:tasks.workunit.client.0.smithi112.stdout: "snapshots": [ 2024-09-13T13:59:01.449 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:01.449 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 1, 2024-09-13T13:59:01.449 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "snap01" 2024-09-13T13:59:01.449 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T13:59:01.449 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:01.449 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 2, 2024-09-13T13:59:01.449 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "snap02" 2024-09-13T13:59:01.449 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T13:59:01.449 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T13:59:01.450 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 5, 2024-09-13T13:59:01.450 INFO:tasks.workunit.client.0.smithi112.stdout: "overlaps": [] 2024-09-13T13:59:01.450 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T13:59:01.450 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:01.450 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 3, 2024-09-13T13:59:01.450 INFO:tasks.workunit.client.0.smithi112.stdout: "snapshots": [ 2024-09-13T13:59:01.450 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:01.450 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 3, 2024-09-13T13:59:01.450 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "snap13" 2024-09-13T13:59:01.450 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T13:59:01.451 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T13:59:01.451 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 6, 2024-09-13T13:59:01.451 INFO:tasks.workunit.client.0.smithi112.stdout: "overlaps": [] 2024-09-13T13:59:01.451 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T13:59:01.451 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:01.451 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 5, 2024-09-13T13:59:01.451 INFO:tasks.workunit.client.0.smithi112.stdout: "snapshots": [ 2024-09-13T13:59:01.451 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:01.451 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 4, 2024-09-13T13:59:01.451 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "snap24" 2024-09-13T13:59:01.451 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T13:59:01.451 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:01.451 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 5, 2024-09-13T13:59:01.451 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "snap25" 2024-09-13T13:59:01.452 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T13:59:01.452 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T13:59:01.452 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 6, 2024-09-13T13:59:01.452 INFO:tasks.workunit.client.0.smithi112.stdout: "overlaps": [] 2024-09-13T13:59:01.452 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T13:59:01.452 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:01.452 INFO:tasks.workunit.client.0.smithi112.stdout: "id": "head", 2024-09-13T13:59:01.452 INFO:tasks.workunit.client.0.smithi112.stdout: "snapshots": [], 2024-09-13T13:59:01.452 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 6 2024-09-13T13:59:01.452 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T13:59:01.452 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T13:59:01.452 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T13:59:01.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-09-13T13:59:01.575 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T13:59:01.575 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T13:59:01.575 INFO:tasks.workunit.client.0.smithi112.stdout: "must": true, 2024-09-13T13:59:01.575 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "0.000000" 2024-09-13T13:59:01.575 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T13:59:01.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-09-13T13:59:04.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-13T13:59:04.894 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T13:59:04.894 INFO:tasks.workunit.client.0.smithi112.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-09-13T13:58:52.153075+0000 28'7 28:37 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T13:59:04.894 INFO:tasks.workunit.client.0.smithi112.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T13:58:52.154762+0000 0'0 28:35 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T13:59:04.894 INFO:tasks.workunit.client.0.smithi112.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T13:58:51.780156+0000 0'0 26:28 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T13:59:04.894 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T13:58:52.153670+0000 0'0 28:33 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T13:59:04.895 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:04.895 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T13:59:04.895 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T13:59:04.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-09-13T13:59:04.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-13T13:59:05.199 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T13:59:05.199 INFO:tasks.workunit.client.0.smithi112.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-09-13T13:58:52.153075+0000 28'7 28:37 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T13:59:05.199 INFO:tasks.workunit.client.0.smithi112.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T13:58:52.154762+0000 0'0 28:35 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T13:59:05.200 INFO:tasks.workunit.client.0.smithi112.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T13:58:51.780156+0000 0'0 26:28 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T13:59:05.200 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T13:58:52.153670+0000 0'0 28:33 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T13:59:05.200 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:05.200 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T13:59:05.200 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T13:59:05.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-09-13T13:59:05.626 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T13:59:05.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-09-13T13:59:06.061 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T13:59:06.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-09-13T13:59:11.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-09-13T13:59:11.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-09-13T13:59:11.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T13:59:11.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T13:59:11.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T13:59:11.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T13:59:11.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T13:59:12.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T13:59:12.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-13T13:59:12.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-09-13T13:59:12.428 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.427+0000 7fb418de9ac0 1 bdev(0x5561d1eb5800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-13T13:59:12.428 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.427+0000 7fb418de9ac0 1 bdev(0x5561d1eb5800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:12.445 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.443+0000 7fb418de9ac0 1 bdev(0x5561d1eb5800 td/osd-mapper/0/block) close 2024-09-13T13:59:12.708 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.707+0000 7fb418de9ac0 1 bdev(0x5561d1eb5800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-13T13:59:12.708 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.707+0000 7fb418de9ac0 1 bdev(0x5561d1eb5800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:12.721 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.719+0000 7fb418de9ac0 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-13T13:59:12.721 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.719+0000 7fb418de9ac0 1 bdev(0x5561d2200000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-13T13:59:12.721 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.719+0000 7fb418de9ac0 1 bdev(0x5561d2200000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:12.721 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.719+0000 7fb418de9ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-13T13:59:12.721 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.719+0000 7fb418de9ac0 1 bluefs mount 2024-09-13T13:59:12.722 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.719+0000 7fb418de9ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-13T13:59:12.730 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.727+0000 7fb418de9ac0 1 bluefs mount shared_bdev_used = 0 2024-09-13T13:59:12.730 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.727+0000 7fb418de9ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-13T13:59:12.791 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.787+0000 7fb418de9ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-13T13:59:12.791 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.787+0000 7fb418de9ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-09-13T13:59:12.791 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.787+0000 7fb418de9ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-09-13T13:59:12.791 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.787+0000 7fb418de9ac0 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-13T13:59:12.791 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.787+0000 7fb418de9ac0 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-09-13T13:59:12.791 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.787+0000 7fb418de9ac0 1 freelist init 2024-09-13T13:59:12.791 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.787+0000 7fb418de9ac0 1 freelist _read_cfg 2024-09-13T13:59:12.791 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.787+0000 7fb418de9ac0 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-13T13:59:12.792 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.791+0000 7fb418de9ac0 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 25 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 9.1553e-07 2024-09-13T13:59:12.792 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.791+0000 7fb418de9ac0 1 bluefs umount 2024-09-13T13:59:12.792 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:12.791+0000 7fb418de9ac0 1 bdev(0x5561d2200000 td/osd-mapper/0/block) close 2024-09-13T13:59:13.016 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.015+0000 7fb418de9ac0 1 bdev(0x5561d2200000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-13T13:59:13.016 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.015+0000 7fb418de9ac0 1 bdev(0x5561d2200000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:13.016 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.015+0000 7fb418de9ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-13T13:59:13.016 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.015+0000 7fb418de9ac0 1 bluefs mount 2024-09-13T13:59:13.025 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.023+0000 7fb418de9ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-13T13:59:13.039 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.035+0000 7fb418de9ac0 1 bluefs mount shared_bdev_used = 27459584 2024-09-13T13:59:13.039 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.035+0000 7fb418de9ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-13T13:59:13.086 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.083+0000 7fb418de9ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-13T13:59:13.087 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-13T13:59:13.087 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 0b 05 58 02 00 00 e1 ae 25 1f 40 8a 40 01 9a a9 |..X.....%.@.@...| 2024-09-13T13:59:13.087 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 ee 40 43 23 4b a2 00 00 00 00 1e 00 00 00 00 00 |.@C#K...........| 2024-09-13T13:59:13.087 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-13T13:59:13.087 INFO:tasks.workunit.client.0.smithi112.stdout:* 2024-09-13T13:59:13.087 INFO:tasks.workunit.client.0.smithi112.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-09-13T13:59:13.087 INFO:tasks.workunit.client.0.smithi112.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-09-13T13:59:13.087 INFO:tasks.workunit.client.0.smithi112.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-09-13T13:59:13.087 INFO:tasks.workunit.client.0.smithi112.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-09-13T13:59:13.087 INFO:tasks.workunit.client.0.smithi112.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-09-13T13:59:13.087 INFO:tasks.workunit.client.0.smithi112.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-09-13T13:59:13.088 INFO:tasks.workunit.client.0.smithi112.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-09-13T13:59:13.088 INFO:tasks.workunit.client.0.smithi112.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-09-13T13:59:13.088 INFO:tasks.workunit.client.0.smithi112.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-09-13T13:59:13.088 INFO:tasks.workunit.client.0.smithi112.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-09-13T13:59:13.088 INFO:tasks.workunit.client.0.smithi112.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-09-13T13:59:13.088 INFO:tasks.workunit.client.0.smithi112.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-09-13T13:59:13.088 INFO:tasks.workunit.client.0.smithi112.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-09-13T13:59:13.088 INFO:tasks.workunit.client.0.smithi112.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-09-13T13:59:13.088 INFO:tasks.workunit.client.0.smithi112.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-09-13T13:59:13.088 INFO:tasks.workunit.client.0.smithi112.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-09-13T13:59:13.088 INFO:tasks.workunit.client.0.smithi112.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-09-13T13:59:13.088 INFO:tasks.workunit.client.0.smithi112.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-09-13T13:59:13.088 INFO:tasks.workunit.client.0.smithi112.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-09-13T13:59:13.089 INFO:tasks.workunit.client.0.smithi112.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-09-13T13:59:13.089 INFO:tasks.workunit.client.0.smithi112.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-09-13T13:59:13.089 INFO:tasks.workunit.client.0.smithi112.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-09-13T13:59:13.089 INFO:tasks.workunit.client.0.smithi112.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-09-13T13:59:13.089 INFO:tasks.workunit.client.0.smithi112.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-09-13T13:59:13.089 INFO:tasks.workunit.client.0.smithi112.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-09-13T13:59:13.089 INFO:tasks.workunit.client.0.smithi112.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-09-13T13:59:13.089 INFO:tasks.workunit.client.0.smithi112.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-09-13T13:59:13.089 INFO:tasks.workunit.client.0.smithi112.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-09-13T13:59:13.089 INFO:tasks.workunit.client.0.smithi112.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-09-13T13:59:13.089 INFO:tasks.workunit.client.0.smithi112.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-09-13T13:59:13.089 INFO:tasks.workunit.client.0.smithi112.stdout:00000220 72 65 1e 00 00 00 05 00 00 00 0b 3f 1c 54 70 83 |re.........?.Tp.| 2024-09-13T13:59:13.089 INFO:tasks.workunit.client.0.smithi112.stdout:00000230 45 c9 9b e6 5d 59 b7 dc 95 d3 00 00 00 00 00 00 |E...]Y..........| 2024-09-13T13:59:13.089 INFO:tasks.workunit.client.0.smithi112.stdout:00000240 00 00 1e 00 00 00 d4 44 e4 66 26 6f 50 2e 01 00 |.......D.f&oP...| 2024-09-13T13:59:13.090 INFO:tasks.workunit.client.0.smithi112.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-09-13T13:59:13.090 INFO:tasks.workunit.client.0.smithi112.stdout:0000025e 2024-09-13T13:59:13.090 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:13.090 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-09-13T13:59:13.090 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:13.090 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:13.090 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-13T13:59:13.090 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-09-13T13:59:13.090 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-13T13:59:13.090 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-09-13T13:59:13.090 INFO:tasks.workunit.client.0.smithi112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-13T13:59:13.090 INFO:tasks.workunit.client.0.smithi112.stdout:00000047 2024-09-13T13:59:13.090 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:13.091 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:13.091 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-09-13T13:59:13.091 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-09-13T13:59:13.091 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-13T13:59:13.091 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-09-13T13:59:13.091 INFO:tasks.workunit.client.0.smithi112.stdout:0000003f 2024-09-13T13:59:13.091 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:13.091 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:13.091 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-13T13:59:13.091 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-09-13T13:59:13.091 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-13T13:59:13.091 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-09-13T13:59:13.091 INFO:tasks.workunit.client.0.smithi112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-13T13:59:13.091 INFO:tasks.workunit.client.0.smithi112.stdout:00000047 2024-09-13T13:59:13.092 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:13.092 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:13.092 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:13.092 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:13.092 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:13.092 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:13.092 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:13.092 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:13.092 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:13.092 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:13.092 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:13.092 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:13.093 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:13.093 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:13.093 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:13.093 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:13.093 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:13.093 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:13.093 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:13.093 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:13.093 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:13.093 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:13.093 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:13.093 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:13.093 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:13.093 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:13.094 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:13.094 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:13.094 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:13.094 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:13.094 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:13.094 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:13.094 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:13.094 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:13.094 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:13.094 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:13.094 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-09-13T13:59:13.094 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:13.094 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-09-13T13:59:13.094 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:13.095 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.083+0000 7fb418de9ac0 1 bluefs umount 2024-09-13T13:59:13.095 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.083+0000 7fb418de9ac0 1 bdev(0x5561d2200000 td/osd-mapper/0/block) close 2024-09-13T13:59:13.304 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.303+0000 7fb418de9ac0 1 freelist shutdown 2024-09-13T13:59:13.304 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.303+0000 7fb418de9ac0 1 bdev(0x5561d1eb5800 td/osd-mapper/0/block) close 2024-09-13T13:59:13.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-13T13:59:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-13T13:59:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-09-13T13:59:13.525 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.523+0000 7ffb95c50ac0 1 bdev(0x55c38bfad800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-13T13:59:13.525 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.523+0000 7ffb95c50ac0 1 bdev(0x55c38bfad800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:13.538 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.535+0000 7ffb95c50ac0 1 bdev(0x55c38bfad800 td/osd-mapper/2/block) close 2024-09-13T13:59:13.812 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.811+0000 7ffb95c50ac0 1 bdev(0x55c38bfad800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-13T13:59:13.812 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.811+0000 7ffb95c50ac0 1 bdev(0x55c38bfad800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:13.813 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.811+0000 7ffb95c50ac0 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-13T13:59:13.813 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.811+0000 7ffb95c50ac0 1 bdev(0x55c38c2f8000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-13T13:59:13.813 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.811+0000 7ffb95c50ac0 1 bdev(0x55c38c2f8000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:13.813 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.811+0000 7ffb95c50ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-13T13:59:13.813 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.811+0000 7ffb95c50ac0 1 bluefs mount 2024-09-13T13:59:13.814 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.811+0000 7ffb95c50ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-13T13:59:13.834 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.831+0000 7ffb95c50ac0 1 bluefs mount shared_bdev_used = 0 2024-09-13T13:59:13.834 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.831+0000 7ffb95c50ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-13T13:59:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.891+0000 7ffb95c50ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-13T13:59:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.891+0000 7ffb95c50ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-09-13T13:59:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.891+0000 7ffb95c50ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-09-13T13:59:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.891+0000 7ffb95c50ac0 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-13T13:59:13.895 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.891+0000 7ffb95c50ac0 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-09-13T13:59:13.895 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.891+0000 7ffb95c50ac0 1 freelist init 2024-09-13T13:59:13.895 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.891+0000 7ffb95c50ac0 1 freelist _read_cfg 2024-09-13T13:59:13.895 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.891+0000 7ffb95c50ac0 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-13T13:59:13.895 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.891+0000 7ffb95c50ac0 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 4.95912e-07 2024-09-13T13:59:13.895 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.891+0000 7ffb95c50ac0 1 bluefs umount 2024-09-13T13:59:13.896 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:13.891+0000 7ffb95c50ac0 1 bdev(0x55c38c2f8000 td/osd-mapper/2/block) close 2024-09-13T13:59:14.100 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:14.099+0000 7ffb95c50ac0 1 bdev(0x55c38c2f8000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-13T13:59:14.100 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:14.099+0000 7ffb95c50ac0 1 bdev(0x55c38c2f8000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:14.100 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:14.099+0000 7ffb95c50ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-13T13:59:14.100 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:14.099+0000 7ffb95c50ac0 1 bluefs mount 2024-09-13T13:59:14.101 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:14.099+0000 7ffb95c50ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-13T13:59:14.107 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:14.103+0000 7ffb95c50ac0 1 bluefs mount shared_bdev_used = 27459584 2024-09-13T13:59:14.107 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:14.103+0000 7ffb95c50ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-13T13:59:14.154 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:14.151+0000 7ffb95c50ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-13T13:59:14.154 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:14.151+0000 7ffb95c50ac0 1 bluefs umount 2024-09-13T13:59:14.154 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:14.151+0000 7ffb95c50ac0 1 bdev(0x55c38c2f8000 td/osd-mapper/2/block) close 2024-09-13T13:59:14.384 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:14.383+0000 7ffb95c50ac0 1 freelist shutdown 2024-09-13T13:59:14.384 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:14.383+0000 7ffb95c50ac0 1 bdev(0x55c38bfad800 td/osd-mapper/2/block) close 2024-09-13T13:59:14.599 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:14.599 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:14.599 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:14.599 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:14.599 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:14.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-13T13:59:14.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-09-13T13:59:14.601 INFO:tasks.workunit.client.0.smithi112.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-09-13T13:59:14.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-13T13:59:14.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-13T13:59:15.255 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-13T13:59:15.256 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 0b 05 58 02 00 00 e1 ae 25 1f 40 8a 40 01 9a a9 |..X.....%.@.@...| 2024-09-13T13:59:15.256 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 ee 40 43 23 4b a2 02 00 00 00 1e 00 00 00 00 00 |.@C#K...........| 2024-09-13T13:59:15.256 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-13T13:59:15.256 INFO:tasks.workunit.client.0.smithi112.stdout:* 2024-09-13T13:59:15.256 INFO:tasks.workunit.client.0.smithi112.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-09-13T13:59:15.256 INFO:tasks.workunit.client.0.smithi112.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-09-13T13:59:15.257 INFO:tasks.workunit.client.0.smithi112.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-09-13T13:59:15.257 INFO:tasks.workunit.client.0.smithi112.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-09-13T13:59:15.257 INFO:tasks.workunit.client.0.smithi112.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-09-13T13:59:15.257 INFO:tasks.workunit.client.0.smithi112.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-09-13T13:59:15.257 INFO:tasks.workunit.client.0.smithi112.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-09-13T13:59:15.257 INFO:tasks.workunit.client.0.smithi112.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-09-13T13:59:15.258 INFO:tasks.workunit.client.0.smithi112.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-09-13T13:59:15.258 INFO:tasks.workunit.client.0.smithi112.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-09-13T13:59:15.258 INFO:tasks.workunit.client.0.smithi112.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-09-13T13:59:15.258 INFO:tasks.workunit.client.0.smithi112.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-09-13T13:59:15.258 INFO:tasks.workunit.client.0.smithi112.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-09-13T13:59:15.258 INFO:tasks.workunit.client.0.smithi112.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-09-13T13:59:15.259 INFO:tasks.workunit.client.0.smithi112.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-09-13T13:59:15.259 INFO:tasks.workunit.client.0.smithi112.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-09-13T13:59:15.259 INFO:tasks.workunit.client.0.smithi112.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-09-13T13:59:15.259 INFO:tasks.workunit.client.0.smithi112.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-09-13T13:59:15.259 INFO:tasks.workunit.client.0.smithi112.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-09-13T13:59:15.260 INFO:tasks.workunit.client.0.smithi112.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-09-13T13:59:15.260 INFO:tasks.workunit.client.0.smithi112.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-09-13T13:59:15.260 INFO:tasks.workunit.client.0.smithi112.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-09-13T13:59:15.260 INFO:tasks.workunit.client.0.smithi112.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-09-13T13:59:15.260 INFO:tasks.workunit.client.0.smithi112.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-09-13T13:59:15.261 INFO:tasks.workunit.client.0.smithi112.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-09-13T13:59:15.261 INFO:tasks.workunit.client.0.smithi112.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-09-13T13:59:15.261 INFO:tasks.workunit.client.0.smithi112.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-09-13T13:59:15.261 INFO:tasks.workunit.client.0.smithi112.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-09-13T13:59:15.261 INFO:tasks.workunit.client.0.smithi112.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-09-13T13:59:15.261 INFO:tasks.workunit.client.0.smithi112.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-09-13T13:59:15.262 INFO:tasks.workunit.client.0.smithi112.stdout:00000220 72 65 1e 00 00 00 0f 00 00 00 46 57 d1 73 9a 8d |re........FW.s..| 2024-09-13T13:59:15.262 INFO:tasks.workunit.client.0.smithi112.stdout:00000230 4d 21 ae ba 8a ad 33 c6 cb 4a 00 00 00 00 00 00 |M!....3..J......| 2024-09-13T13:59:15.262 INFO:tasks.workunit.client.0.smithi112.stdout:00000240 00 00 1e 00 00 00 18 45 e4 66 5a 86 3b 38 01 00 |.......E.fZ.;8..| 2024-09-13T13:59:15.262 INFO:tasks.workunit.client.0.smithi112.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-09-13T13:59:15.262 INFO:tasks.workunit.client.0.smithi112.stdout:0000025e 2024-09-13T13:59:15.262 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:15.262 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-09-13T13:59:15.263 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:15.263 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:15.263 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-13T13:59:15.263 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-09-13T13:59:15.263 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-13T13:59:15.264 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-09-13T13:59:15.264 INFO:tasks.workunit.client.0.smithi112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-13T13:59:15.264 INFO:tasks.workunit.client.0.smithi112.stdout:00000047 2024-09-13T13:59:15.264 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:15.264 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:15.264 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-09-13T13:59:15.265 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-09-13T13:59:15.265 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-13T13:59:15.265 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-09-13T13:59:15.265 INFO:tasks.workunit.client.0.smithi112.stdout:0000003f 2024-09-13T13:59:15.265 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:15.266 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:15.266 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-13T13:59:15.266 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-09-13T13:59:15.266 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-13T13:59:15.266 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-09-13T13:59:15.266 INFO:tasks.workunit.client.0.smithi112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-13T13:59:15.267 INFO:tasks.workunit.client.0.smithi112.stdout:00000047 2024-09-13T13:59:15.267 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:15.267 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:15.267 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:15.267 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:15.267 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:15.268 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:15.268 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:15.268 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:15.268 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:15.268 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:15.269 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:15.269 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:15.269 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:15.269 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:15.269 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:15.269 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:15.270 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:15.270 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:15.270 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:15.270 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:15.270 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:15.270 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:15.271 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:15.271 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:15.271 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:15.271 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:15.271 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:15.271 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:15.272 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:15.272 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:15.272 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:15.272 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:15.272 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:15.272 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:15.273 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:15.273 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:15.273 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-09-13T13:59:15.273 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:15.273 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-09-13T13:59:15.273 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:15.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-13T13:59:15.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-09-13T13:59:15.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-13T13:59:15.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2024-09-13T13:59:16.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2024-09-13T13:59:16.796 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2024-09-13T13:59:16.798 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2024-09-13T13:59:16.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-13T13:59:16.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2024-09-13T13:59:16.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2024-09-13T13:59:16.799 INFO:tasks.workunit.client.0.smithi112.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2024-09-13T13:59:16.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-09-13T13:59:16.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-09-13T13:59:16.828 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:16.827+0000 7fca5bc7fac0 1 bdev(0x563e369a9800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-13T13:59:16.829 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:16.827+0000 7fca5bc7fac0 1 bdev(0x563e369a9800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:16.846 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:16.839+0000 7fca5bc7fac0 1 bdev(0x563e369a9800 td/osd-mapper/0/block) close 2024-09-13T13:59:17.112 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.111+0000 7fca5bc7fac0 1 bdev(0x563e369a9800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-13T13:59:17.112 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.111+0000 7fca5bc7fac0 1 bdev(0x563e369a9800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:17.113 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.111+0000 7fca5bc7fac0 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-13T13:59:17.113 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.111+0000 7fca5bc7fac0 1 bdev(0x563e36cf4000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-13T13:59:17.113 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.111+0000 7fca5bc7fac0 1 bdev(0x563e36cf4000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:17.113 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.111+0000 7fca5bc7fac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-13T13:59:17.113 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.111+0000 7fca5bc7fac0 1 bluefs mount 2024-09-13T13:59:17.114 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.111+0000 7fca5bc7fac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-13T13:59:17.122 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.119+0000 7fca5bc7fac0 1 bluefs mount shared_bdev_used = 0 2024-09-13T13:59:17.122 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.119+0000 7fca5bc7fac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-13T13:59:17.183 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.179+0000 7fca5bc7fac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-13T13:59:17.183 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.179+0000 7fca5bc7fac0 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-09-13T13:59:17.183 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.179+0000 7fca5bc7fac0 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-09-13T13:59:17.183 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.179+0000 7fca5bc7fac0 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-13T13:59:17.183 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.179+0000 7fca5bc7fac0 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-09-13T13:59:17.183 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.179+0000 7fca5bc7fac0 1 freelist init 2024-09-13T13:59:17.183 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.179+0000 7fca5bc7fac0 1 freelist _read_cfg 2024-09-13T13:59:17.183 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.179+0000 7fca5bc7fac0 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-13T13:59:17.184 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.179+0000 7fca5bc7fac0 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 25 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 9.1553e-07 2024-09-13T13:59:17.184 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.183+0000 7fca5bc7fac0 1 bluefs umount 2024-09-13T13:59:17.184 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.183+0000 7fca5bc7fac0 1 bdev(0x563e36cf4000 td/osd-mapper/0/block) close 2024-09-13T13:59:17.396 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.395+0000 7fca5bc7fac0 1 bdev(0x563e36cf4000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-13T13:59:17.396 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.395+0000 7fca5bc7fac0 1 bdev(0x563e36cf4000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:17.396 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.395+0000 7fca5bc7fac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-13T13:59:17.396 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.395+0000 7fca5bc7fac0 1 bluefs mount 2024-09-13T13:59:17.407 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.407+0000 7fca5bc7fac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-13T13:59:17.417 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.415+0000 7fca5bc7fac0 1 bluefs mount shared_bdev_used = 27459584 2024-09-13T13:59:17.417 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.415+0000 7fca5bc7fac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-13T13:59:17.463 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.459+0000 7fca5bc7fac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-13T13:59:17.464 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-13T13:59:17.464 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 0b 05 58 02 00 00 e1 ae 25 1f 40 8a 40 01 9a a9 |..X.....%.@.@...| 2024-09-13T13:59:17.464 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 ee 40 43 23 4b a2 00 00 00 00 1e 00 00 00 00 00 |.@C#K...........| 2024-09-13T13:59:17.464 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-13T13:59:17.464 INFO:tasks.workunit.client.0.smithi112.stdout:* 2024-09-13T13:59:17.464 INFO:tasks.workunit.client.0.smithi112.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-09-13T13:59:17.464 INFO:tasks.workunit.client.0.smithi112.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-09-13T13:59:17.465 INFO:tasks.workunit.client.0.smithi112.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-09-13T13:59:17.465 INFO:tasks.workunit.client.0.smithi112.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-09-13T13:59:17.465 INFO:tasks.workunit.client.0.smithi112.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-09-13T13:59:17.465 INFO:tasks.workunit.client.0.smithi112.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-09-13T13:59:17.465 INFO:tasks.workunit.client.0.smithi112.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-09-13T13:59:17.465 INFO:tasks.workunit.client.0.smithi112.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-09-13T13:59:17.465 INFO:tasks.workunit.client.0.smithi112.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-09-13T13:59:17.465 INFO:tasks.workunit.client.0.smithi112.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-09-13T13:59:17.465 INFO:tasks.workunit.client.0.smithi112.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-09-13T13:59:17.465 INFO:tasks.workunit.client.0.smithi112.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-09-13T13:59:17.465 INFO:tasks.workunit.client.0.smithi112.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-09-13T13:59:17.465 INFO:tasks.workunit.client.0.smithi112.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-09-13T13:59:17.465 INFO:tasks.workunit.client.0.smithi112.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-09-13T13:59:17.466 INFO:tasks.workunit.client.0.smithi112.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-09-13T13:59:17.466 INFO:tasks.workunit.client.0.smithi112.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-09-13T13:59:17.466 INFO:tasks.workunit.client.0.smithi112.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-09-13T13:59:17.466 INFO:tasks.workunit.client.0.smithi112.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-09-13T13:59:17.466 INFO:tasks.workunit.client.0.smithi112.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-09-13T13:59:17.466 INFO:tasks.workunit.client.0.smithi112.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-09-13T13:59:17.466 INFO:tasks.workunit.client.0.smithi112.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-09-13T13:59:17.466 INFO:tasks.workunit.client.0.smithi112.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-09-13T13:59:17.466 INFO:tasks.workunit.client.0.smithi112.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-09-13T13:59:17.466 INFO:tasks.workunit.client.0.smithi112.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-09-13T13:59:17.466 INFO:tasks.workunit.client.0.smithi112.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-09-13T13:59:17.466 INFO:tasks.workunit.client.0.smithi112.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-09-13T13:59:17.466 INFO:tasks.workunit.client.0.smithi112.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-09-13T13:59:17.466 INFO:tasks.workunit.client.0.smithi112.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-09-13T13:59:17.467 INFO:tasks.workunit.client.0.smithi112.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-09-13T13:59:17.467 INFO:tasks.workunit.client.0.smithi112.stdout:00000220 72 65 1e 00 00 00 05 00 00 00 0b 3f 1c 54 70 83 |re.........?.Tp.| 2024-09-13T13:59:17.467 INFO:tasks.workunit.client.0.smithi112.stdout:00000230 45 c9 9b e6 5d 59 b7 dc 95 d3 00 00 00 00 00 00 |E...]Y..........| 2024-09-13T13:59:17.467 INFO:tasks.workunit.client.0.smithi112.stdout:00000240 00 00 1e 00 00 00 d4 44 e4 66 26 6f 50 2e 01 00 |.......D.f&oP...| 2024-09-13T13:59:17.467 INFO:tasks.workunit.client.0.smithi112.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-09-13T13:59:17.467 INFO:tasks.workunit.client.0.smithi112.stdout:0000025e 2024-09-13T13:59:17.467 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:17.467 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-09-13T13:59:17.467 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:17.467 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:17.467 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-13T13:59:17.467 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-09-13T13:59:17.467 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-13T13:59:17.468 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-09-13T13:59:17.468 INFO:tasks.workunit.client.0.smithi112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-13T13:59:17.468 INFO:tasks.workunit.client.0.smithi112.stdout:00000047 2024-09-13T13:59:17.468 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:17.468 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:17.468 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-09-13T13:59:17.468 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-09-13T13:59:17.468 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-13T13:59:17.468 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-09-13T13:59:17.468 INFO:tasks.workunit.client.0.smithi112.stdout:0000003f 2024-09-13T13:59:17.468 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:17.468 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:17.468 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-13T13:59:17.468 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-09-13T13:59:17.469 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-13T13:59:17.469 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-09-13T13:59:17.469 INFO:tasks.workunit.client.0.smithi112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-13T13:59:17.469 INFO:tasks.workunit.client.0.smithi112.stdout:00000047 2024-09-13T13:59:17.469 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:17.469 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:17.469 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:17.469 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:17.469 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:17.469 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:17.469 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:17.469 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:17.469 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:17.470 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:17.470 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:17.470 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:17.470 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:17.470 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:17.470 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:17.470 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:17.470 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:17.470 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:17.470 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:17.470 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:17.470 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:17.470 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:17.470 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:17.471 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:17.471 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:17.471 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:17.471 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:17.471 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:17.471 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:17.471 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:17.471 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:17.471 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:17.471 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:17.471 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:17.471 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:17.471 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:17.472 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-09-13T13:59:17.472 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:17.472 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-09-13T13:59:17.472 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:17.472 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.459+0000 7fca5bc7fac0 1 bluefs umount 2024-09-13T13:59:17.472 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.459+0000 7fca5bc7fac0 1 bdev(0x563e36cf4000 td/osd-mapper/0/block) close 2024-09-13T13:59:17.684 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.683+0000 7fca5bc7fac0 1 freelist shutdown 2024-09-13T13:59:17.684 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:17.683+0000 7fca5bc7fac0 1 bdev(0x563e369a9800 td/osd-mapper/0/block) close 2024-09-13T13:59:17.899 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-09-13T13:59:17.900 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-09-13T13:59:17.900 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-13T13:59:19.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:19.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-13T13:59:19.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-09-13T13:59:19.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-09-13T13:59:19.004 INFO:tasks.workunit.client.0.smithi112.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:19.005 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-09-13T13:59:19.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.Y2Z9MOXlxa_the_val 2024-09-13T13:59:19.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-13T13:59:19.007 INFO:tasks.workunit.client.0.smithi112.stdout:Value dumped in: /tmp/tmp.Y2Z9MOXlxa_the_val 2024-09-13T13:59:19.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.Y2Z9MOXlxa_the_val' 2024-09-13T13:59:19.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.Y2Z9MOXlxa_the_val 2024-09-13T13:59:19.672 INFO:tasks.workunit.client.0.smithi112.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-09-13T13:59:20.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-13T13:59:20.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.Y2Z9MOXlxa_the_val 2024-09-13T13:59:20.122 INFO:tasks.workunit.client.0.smithi112.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-09-13T13:59:20.122 INFO:tasks.workunit.client.0.smithi112.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-09-13T13:59:20.122 INFO:tasks.workunit.client.0.smithi112.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-09-13T13:59:20.122 INFO:tasks.workunit.client.0.smithi112.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-09-13T13:59:20.122 INFO:tasks.workunit.client.0.smithi112.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-09-13T13:59:20.122 INFO:tasks.workunit.client.0.smithi112.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-09-13T13:59:20.122 INFO:tasks.workunit.client.0.smithi112.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-09-13T13:59:20.122 INFO:tasks.workunit.client.0.smithi112.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-09-13T13:59:20.122 INFO:tasks.workunit.client.0.smithi112.stdout:0000073 2024-09-13T13:59:20.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-13T13:59:20.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:21.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.Y2Z9MOXlxa_the_val 2024-09-13T13:59:23.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-13T13:59:23.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-13T13:59:23.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-09-13T13:59:23.342 INFO:tasks.workunit.client.0.smithi112.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-09-13T13:59:23.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-09-13T13:59:23.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-09-13T13:59:23.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-13T13:59:23.371 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.367+0000 7fe18ab37ac0 1 bdev(0x5623f2e07800 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-13T13:59:23.371 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.367+0000 7fe18ab37ac0 1 bdev(0x5623f2e07800 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:23.381 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.379+0000 7fe18ab37ac0 1 bdev(0x5623f2e07800 td/osd-mapper/1/block) close 2024-09-13T13:59:23.652 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.650+0000 7fe18ab37ac0 1 bdev(0x5623f2e07800 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-13T13:59:23.652 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.650+0000 7fe18ab37ac0 1 bdev(0x5623f2e07800 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:23.653 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.650+0000 7fe18ab37ac0 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-13T13:59:23.653 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.650+0000 7fe18ab37ac0 1 bdev(0x5623f3152000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-13T13:59:23.653 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.650+0000 7fe18ab37ac0 1 bdev(0x5623f3152000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:23.653 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.650+0000 7fe18ab37ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-09-13T13:59:23.653 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.650+0000 7fe18ab37ac0 1 bluefs mount 2024-09-13T13:59:23.653 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.650+0000 7fe18ab37ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-13T13:59:23.661 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.658+0000 7fe18ab37ac0 1 bluefs mount shared_bdev_used = 0 2024-09-13T13:59:23.661 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.658+0000 7fe18ab37ac0 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-13T13:59:23.717 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.714+0000 7fe18ab37ac0 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-13T13:59:23.717 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.714+0000 7fe18ab37ac0 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-09-13T13:59:23.717 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.714+0000 7fe18ab37ac0 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-09-13T13:59:23.717 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.714+0000 7fe18ab37ac0 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-13T13:59:23.718 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.714+0000 7fe18ab37ac0 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-09-13T13:59:23.718 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.714+0000 7fe18ab37ac0 1 freelist init 2024-09-13T13:59:23.718 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.714+0000 7fe18ab37ac0 1 freelist _read_cfg 2024-09-13T13:59:23.718 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.714+0000 7fe18ab37ac0 1 bluestore(td/osd-mapper/1) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-13T13:59:23.718 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.714+0000 7fe18ab37ac0 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 17 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 6.10353e-07 2024-09-13T13:59:23.718 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.714+0000 7fe18ab37ac0 1 bluefs umount 2024-09-13T13:59:23.719 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.714+0000 7fe18ab37ac0 1 bdev(0x5623f3152000 td/osd-mapper/1/block) close 2024-09-13T13:59:23.932 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.930+0000 7fe18ab37ac0 1 bdev(0x5623f3152000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-13T13:59:23.932 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.930+0000 7fe18ab37ac0 1 bdev(0x5623f3152000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:23.932 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.930+0000 7fe18ab37ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-09-13T13:59:23.932 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.930+0000 7fe18ab37ac0 1 bluefs mount 2024-09-13T13:59:23.938 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.934+0000 7fe18ab37ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-13T13:59:23.947 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.942+0000 7fe18ab37ac0 1 bluefs mount shared_bdev_used = 27459584 2024-09-13T13:59:23.947 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.942+0000 7fe18ab37ac0 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-13T13:59:23.994 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.990+0000 7fe18ab37ac0 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-13T13:59:23.995 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-13T13:59:23.996 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 0b 05 58 02 00 00 e1 ae 25 1f 40 8a 40 01 9a a9 |..X.....%.@.@...| 2024-09-13T13:59:23.996 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 ee 40 43 23 4b a2 01 00 00 00 1e 00 00 00 00 00 |.@C#K...........| 2024-09-13T13:59:23.996 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-13T13:59:23.996 INFO:tasks.workunit.client.0.smithi112.stdout:* 2024-09-13T13:59:23.996 INFO:tasks.workunit.client.0.smithi112.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-09-13T13:59:23.996 INFO:tasks.workunit.client.0.smithi112.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-09-13T13:59:23.996 INFO:tasks.workunit.client.0.smithi112.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-09-13T13:59:23.997 INFO:tasks.workunit.client.0.smithi112.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-09-13T13:59:23.997 INFO:tasks.workunit.client.0.smithi112.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-09-13T13:59:23.997 INFO:tasks.workunit.client.0.smithi112.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-09-13T13:59:23.997 INFO:tasks.workunit.client.0.smithi112.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-09-13T13:59:23.997 INFO:tasks.workunit.client.0.smithi112.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-09-13T13:59:23.997 INFO:tasks.workunit.client.0.smithi112.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-09-13T13:59:23.997 INFO:tasks.workunit.client.0.smithi112.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-09-13T13:59:23.997 INFO:tasks.workunit.client.0.smithi112.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-09-13T13:59:23.998 INFO:tasks.workunit.client.0.smithi112.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-09-13T13:59:23.998 INFO:tasks.workunit.client.0.smithi112.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-09-13T13:59:23.998 INFO:tasks.workunit.client.0.smithi112.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-09-13T13:59:23.998 INFO:tasks.workunit.client.0.smithi112.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-09-13T13:59:23.998 INFO:tasks.workunit.client.0.smithi112.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-09-13T13:59:23.998 INFO:tasks.workunit.client.0.smithi112.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-09-13T13:59:23.999 INFO:tasks.workunit.client.0.smithi112.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-09-13T13:59:23.999 INFO:tasks.workunit.client.0.smithi112.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-09-13T13:59:23.999 INFO:tasks.workunit.client.0.smithi112.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-09-13T13:59:23.999 INFO:tasks.workunit.client.0.smithi112.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-09-13T13:59:23.999 INFO:tasks.workunit.client.0.smithi112.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-09-13T13:59:24.000 INFO:tasks.workunit.client.0.smithi112.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-09-13T13:59:24.000 INFO:tasks.workunit.client.0.smithi112.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-09-13T13:59:24.000 INFO:tasks.workunit.client.0.smithi112.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-09-13T13:59:24.000 INFO:tasks.workunit.client.0.smithi112.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-09-13T13:59:24.000 INFO:tasks.workunit.client.0.smithi112.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-09-13T13:59:24.000 INFO:tasks.workunit.client.0.smithi112.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-09-13T13:59:24.000 INFO:tasks.workunit.client.0.smithi112.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-09-13T13:59:24.000 INFO:tasks.workunit.client.0.smithi112.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-09-13T13:59:24.000 INFO:tasks.workunit.client.0.smithi112.stdout:00000220 72 65 1e 00 00 00 0a 00 00 00 d4 e5 e5 93 57 4b |re............WK| 2024-09-13T13:59:24.001 INFO:tasks.workunit.client.0.smithi112.stdout:00000230 4d bd a0 48 ee 4e 8d 28 33 96 00 00 00 00 00 00 |M..H.N.(3.......| 2024-09-13T13:59:24.001 INFO:tasks.workunit.client.0.smithi112.stdout:00000240 00 00 1e 00 00 00 f6 44 e4 66 53 7a c7 1b 01 00 |.......D.fSz....| 2024-09-13T13:59:24.001 INFO:tasks.workunit.client.0.smithi112.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-09-13T13:59:24.001 INFO:tasks.workunit.client.0.smithi112.stdout:0000025e 2024-09-13T13:59:24.001 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:24.001 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-09-13T13:59:24.001 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:24.001 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:24.002 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-13T13:59:24.002 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-09-13T13:59:24.002 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-13T13:59:24.002 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-09-13T13:59:24.002 INFO:tasks.workunit.client.0.smithi112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-13T13:59:24.002 INFO:tasks.workunit.client.0.smithi112.stdout:00000047 2024-09-13T13:59:24.002 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:24.002 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:24.003 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-09-13T13:59:24.003 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-09-13T13:59:24.003 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-13T13:59:24.003 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-09-13T13:59:24.003 INFO:tasks.workunit.client.0.smithi112.stdout:0000003f 2024-09-13T13:59:24.003 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:24.003 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:24.003 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-13T13:59:24.003 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-09-13T13:59:24.004 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-13T13:59:24.004 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-09-13T13:59:24.004 INFO:tasks.workunit.client.0.smithi112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-13T13:59:24.004 INFO:tasks.workunit.client.0.smithi112.stdout:00000047 2024-09-13T13:59:24.004 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:24.004 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:24.004 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:24.004 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:24.005 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:24.005 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:24.005 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:24.005 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:24.005 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:24.005 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:24.005 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:24.005 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:24.006 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:24.006 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:24.006 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:24.006 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:24.006 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:24.006 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:24.006 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:24.006 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:24.006 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:24.007 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:24.007 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:24.007 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:24.007 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:24.007 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:24.007 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:24.007 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:24.007 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:24.008 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:24.008 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:24.008 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:24.008 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:24.008 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:24.008 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:24.008 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:24.008 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-09-13T13:59:24.008 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:24.008 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-09-13T13:59:24.009 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:24.009 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.990+0000 7fe18ab37ac0 1 bluefs umount 2024-09-13T13:59:24.009 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:23.990+0000 7fe18ab37ac0 1 bdev(0x5623f3152000 td/osd-mapper/1/block) close 2024-09-13T13:59:24.236 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:24.234+0000 7fe18ab37ac0 1 freelist shutdown 2024-09-13T13:59:24.236 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:24.234+0000 7fe18ab37ac0 1 bdev(0x5623f2e07800 td/osd-mapper/1/block) close 2024-09-13T13:59:24.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-13T13:59:24.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-09-13T13:59:24.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-13T13:59:25.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:25.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-13T13:59:25.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-09-13T13:59:25.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-09-13T13:59:25.544 INFO:tasks.workunit.client.0.smithi112.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:25.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-09-13T13:59:25.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.o0wpUxW5ug_the_val 2024-09-13T13:59:25.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-13T13:59:25.547 INFO:tasks.workunit.client.0.smithi112.stdout:Value dumped in: /tmp/tmp.o0wpUxW5ug_the_val 2024-09-13T13:59:25.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.o0wpUxW5ug_the_val' 2024-09-13T13:59:25.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.o0wpUxW5ug_the_val 2024-09-13T13:59:26.205 INFO:tasks.workunit.client.0.smithi112.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-09-13T13:59:26.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-13T13:59:26.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.o0wpUxW5ug_the_val 2024-09-13T13:59:26.658 INFO:tasks.workunit.client.0.smithi112.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-09-13T13:59:26.659 INFO:tasks.workunit.client.0.smithi112.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-09-13T13:59:26.659 INFO:tasks.workunit.client.0.smithi112.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-09-13T13:59:26.659 INFO:tasks.workunit.client.0.smithi112.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-09-13T13:59:26.659 INFO:tasks.workunit.client.0.smithi112.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-09-13T13:59:26.659 INFO:tasks.workunit.client.0.smithi112.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-09-13T13:59:26.659 INFO:tasks.workunit.client.0.smithi112.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-09-13T13:59:26.659 INFO:tasks.workunit.client.0.smithi112.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-09-13T13:59:26.660 INFO:tasks.workunit.client.0.smithi112.stdout:0000073 2024-09-13T13:59:26.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-13T13:59:26.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:28.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.o0wpUxW5ug_the_val 2024-09-13T13:59:29.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-13T13:59:29.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-13T13:59:29.870 INFO:tasks.workunit.client.0.smithi112.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-09-13T13:59:29.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-09-13T13:59:29.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-09-13T13:59:29.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-09-13T13:59:29.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-13T13:59:29.899 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:29.894+0000 7f5060302ac0 1 bdev(0x564e5413d800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-13T13:59:29.899 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:29.894+0000 7f5060302ac0 1 bdev(0x564e5413d800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:29.911 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:29.906+0000 7f5060302ac0 1 bdev(0x564e5413d800 td/osd-mapper/2/block) close 2024-09-13T13:59:30.180 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.178+0000 7f5060302ac0 1 bdev(0x564e5413d800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-13T13:59:30.180 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.178+0000 7f5060302ac0 1 bdev(0x564e5413d800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:30.192 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.186+0000 7f5060302ac0 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-13T13:59:30.192 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.186+0000 7f5060302ac0 1 bdev(0x564e54488000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-13T13:59:30.192 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.186+0000 7f5060302ac0 1 bdev(0x564e54488000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:30.192 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.186+0000 7f5060302ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-13T13:59:30.192 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.186+0000 7f5060302ac0 1 bluefs mount 2024-09-13T13:59:30.192 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.186+0000 7f5060302ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-13T13:59:30.199 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.194+0000 7f5060302ac0 1 bluefs mount shared_bdev_used = 0 2024-09-13T13:59:30.199 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.194+0000 7f5060302ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-13T13:59:30.258 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.254+0000 7f5060302ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-13T13:59:30.258 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.254+0000 7f5060302ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-09-13T13:59:30.258 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.254+0000 7f5060302ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-09-13T13:59:30.259 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.254+0000 7f5060302ac0 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-13T13:59:30.259 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.254+0000 7f5060302ac0 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-09-13T13:59:30.259 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.254+0000 7f5060302ac0 1 freelist init 2024-09-13T13:59:30.259 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.254+0000 7f5060302ac0 1 freelist _read_cfg 2024-09-13T13:59:30.259 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.254+0000 7f5060302ac0 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-13T13:59:30.259 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.254+0000 7f5060302ac0 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbc000, fragmentation 4.95912e-07 2024-09-13T13:59:30.259 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.254+0000 7f5060302ac0 1 bluefs umount 2024-09-13T13:59:30.260 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.254+0000 7f5060302ac0 1 bdev(0x564e54488000 td/osd-mapper/2/block) close 2024-09-13T13:59:30.472 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.470+0000 7f5060302ac0 1 bdev(0x564e54488000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-13T13:59:30.472 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.470+0000 7f5060302ac0 1 bdev(0x564e54488000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-13T13:59:30.472 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.470+0000 7f5060302ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-13T13:59:30.472 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.470+0000 7f5060302ac0 1 bluefs mount 2024-09-13T13:59:30.473 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.470+0000 7f5060302ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-13T13:59:30.479 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.474+0000 7f5060302ac0 1 bluefs mount shared_bdev_used = 27459584 2024-09-13T13:59:30.479 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.474+0000 7f5060302ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-13T13:59:30.527 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.522+0000 7f5060302ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-13T13:59:30.528 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-13T13:59:30.528 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 0b 05 58 02 00 00 e1 ae 25 1f 40 8a 40 01 9a a9 |..X.....%.@.@...| 2024-09-13T13:59:30.528 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 ee 40 43 23 4b a2 02 00 00 00 1e 00 00 00 00 00 |.@C#K...........| 2024-09-13T13:59:30.528 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-09-13T13:59:30.529 INFO:tasks.workunit.client.0.smithi112.stdout:* 2024-09-13T13:59:30.529 INFO:tasks.workunit.client.0.smithi112.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-09-13T13:59:30.529 INFO:tasks.workunit.client.0.smithi112.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-09-13T13:59:30.529 INFO:tasks.workunit.client.0.smithi112.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-09-13T13:59:30.529 INFO:tasks.workunit.client.0.smithi112.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-09-13T13:59:30.529 INFO:tasks.workunit.client.0.smithi112.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-09-13T13:59:30.529 INFO:tasks.workunit.client.0.smithi112.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-09-13T13:59:30.529 INFO:tasks.workunit.client.0.smithi112.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-09-13T13:59:30.529 INFO:tasks.workunit.client.0.smithi112.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-09-13T13:59:30.529 INFO:tasks.workunit.client.0.smithi112.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-09-13T13:59:30.529 INFO:tasks.workunit.client.0.smithi112.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-09-13T13:59:30.529 INFO:tasks.workunit.client.0.smithi112.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-09-13T13:59:30.529 INFO:tasks.workunit.client.0.smithi112.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-09-13T13:59:30.529 INFO:tasks.workunit.client.0.smithi112.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-09-13T13:59:30.530 INFO:tasks.workunit.client.0.smithi112.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-09-13T13:59:30.530 INFO:tasks.workunit.client.0.smithi112.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-09-13T13:59:30.530 INFO:tasks.workunit.client.0.smithi112.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-09-13T13:59:30.530 INFO:tasks.workunit.client.0.smithi112.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-09-13T13:59:30.530 INFO:tasks.workunit.client.0.smithi112.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-09-13T13:59:30.530 INFO:tasks.workunit.client.0.smithi112.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-09-13T13:59:30.530 INFO:tasks.workunit.client.0.smithi112.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-09-13T13:59:30.530 INFO:tasks.workunit.client.0.smithi112.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-09-13T13:59:30.530 INFO:tasks.workunit.client.0.smithi112.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-09-13T13:59:30.530 INFO:tasks.workunit.client.0.smithi112.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-09-13T13:59:30.530 INFO:tasks.workunit.client.0.smithi112.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-09-13T13:59:30.530 INFO:tasks.workunit.client.0.smithi112.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-09-13T13:59:30.530 INFO:tasks.workunit.client.0.smithi112.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-09-13T13:59:30.530 INFO:tasks.workunit.client.0.smithi112.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-09-13T13:59:30.531 INFO:tasks.workunit.client.0.smithi112.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-09-13T13:59:30.531 INFO:tasks.workunit.client.0.smithi112.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-09-13T13:59:30.531 INFO:tasks.workunit.client.0.smithi112.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-09-13T13:59:30.531 INFO:tasks.workunit.client.0.smithi112.stdout:00000220 72 65 1e 00 00 00 0f 00 00 00 46 57 d1 73 9a 8d |re........FW.s..| 2024-09-13T13:59:30.531 INFO:tasks.workunit.client.0.smithi112.stdout:00000230 4d 21 ae ba 8a ad 33 c6 cb 4a 00 00 00 00 00 00 |M!....3..J......| 2024-09-13T13:59:30.531 INFO:tasks.workunit.client.0.smithi112.stdout:00000240 00 00 1e 00 00 00 18 45 e4 66 5a 86 3b 38 01 00 |.......E.fZ.;8..| 2024-09-13T13:59:30.531 INFO:tasks.workunit.client.0.smithi112.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1e 00 00 00 |..............| 2024-09-13T13:59:30.531 INFO:tasks.workunit.client.0.smithi112.stdout:0000025e 2024-09-13T13:59:30.531 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:30.531 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-09-13T13:59:30.531 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:30.531 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:30.531 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-13T13:59:30.532 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-09-13T13:59:30.532 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-13T13:59:30.532 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-09-13T13:59:30.532 INFO:tasks.workunit.client.0.smithi112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-13T13:59:30.532 INFO:tasks.workunit.client.0.smithi112.stdout:00000047 2024-09-13T13:59:30.532 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:30.532 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:30.532 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-09-13T13:59:30.532 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-09-13T13:59:30.532 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-13T13:59:30.532 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-09-13T13:59:30.532 INFO:tasks.workunit.client.0.smithi112.stdout:0000003f 2024-09-13T13:59:30.532 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:30.532 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:30.533 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-09-13T13:59:30.533 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-09-13T13:59:30.533 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-09-13T13:59:30.533 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-09-13T13:59:30.533 INFO:tasks.workunit.client.0.smithi112.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-09-13T13:59:30.533 INFO:tasks.workunit.client.0.smithi112.stdout:00000047 2024-09-13T13:59:30.533 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:30.533 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:30.533 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:30.533 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:30.533 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:30.533 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:30.533 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:30.533 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:30.534 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T13:59:30.534 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:30.534 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:30.534 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:30.534 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:30.534 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:30.534 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:30.534 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:30.534 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:30.534 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:30.534 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:30.534 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:30.535 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:30.535 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:30.535 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:30.535 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:30.535 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:30.535 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:30.535 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:30.535 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:30.535 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:30.535 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T13:59:30.535 INFO:tasks.workunit.client.0.smithi112.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-09-13T13:59:30.536 INFO:tasks.workunit.client.0.smithi112.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-09-13T13:59:30.536 INFO:tasks.workunit.client.0.smithi112.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-09-13T13:59:30.536 INFO:tasks.workunit.client.0.smithi112.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-09-13T13:59:30.536 INFO:tasks.workunit.client.0.smithi112.stdout:0000003b 2024-09-13T13:59:30.536 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:30.536 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-09-13T13:59:30.536 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:30.536 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-09-13T13:59:30.536 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:30.536 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.522+0000 7f5060302ac0 1 bluefs umount 2024-09-13T13:59:30.536 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.522+0000 7f5060302ac0 1 bdev(0x564e54488000 td/osd-mapper/2/block) close 2024-09-13T13:59:30.752 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.750+0000 7f5060302ac0 1 freelist shutdown 2024-09-13T13:59:30.752 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:30.750+0000 7f5060302ac0 1 bdev(0x564e5413d800 td/osd-mapper/2/block) close 2024-09-13T13:59:30.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-13T13:59:30.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-09-13T13:59:30.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-13T13:59:32.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:32.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-13T13:59:32.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-09-13T13:59:32.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-09-13T13:59:32.073 INFO:tasks.workunit.client.0.smithi112.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:32.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-09-13T13:59:32.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.T5jpLgKK8i_the_val 2024-09-13T13:59:32.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-13T13:59:32.075 INFO:tasks.workunit.client.0.smithi112.stdout:Value dumped in: /tmp/tmp.T5jpLgKK8i_the_val 2024-09-13T13:59:32.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.T5jpLgKK8i_the_val' 2024-09-13T13:59:32.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.T5jpLgKK8i_the_val 2024-09-13T13:59:32.743 INFO:tasks.workunit.client.0.smithi112.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-09-13T13:59:33.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-13T13:59:33.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.T5jpLgKK8i_the_val 2024-09-13T13:59:33.187 INFO:tasks.workunit.client.0.smithi112.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-09-13T13:59:33.187 INFO:tasks.workunit.client.0.smithi112.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-09-13T13:59:33.187 INFO:tasks.workunit.client.0.smithi112.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-09-13T13:59:33.187 INFO:tasks.workunit.client.0.smithi112.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-09-13T13:59:33.187 INFO:tasks.workunit.client.0.smithi112.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-09-13T13:59:33.187 INFO:tasks.workunit.client.0.smithi112.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-09-13T13:59:33.187 INFO:tasks.workunit.client.0.smithi112.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-09-13T13:59:33.188 INFO:tasks.workunit.client.0.smithi112.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-09-13T13:59:33.188 INFO:tasks.workunit.client.0.smithi112.stdout:0000073 2024-09-13T13:59:33.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-13T13:59:33.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T13:59:35.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.T5jpLgKK8i_the_val 2024-09-13T13:59:36.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-13T13:59:36.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 ' 2024-09-13T13:59:36.422 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 2024-09-13T13:59:36.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T13:59:36.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-13T13:59:36.423 INFO:tasks.workunit.client.0.smithi112.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ 2024-09-13T13:59:36.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/' 2024-09-13T13:59:36.424 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-09-13T13:59:36.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-09-13T13:59:36.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-13T13:59:36.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=e1ae251f-408a-4001-9aa9-ee4043234ba2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T13:59:36.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-09-13T13:59:36.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2024-09-13T13:59:36.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T13:59:36.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T13:59:36.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T13:59:36.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/0 2024-09-13T13:59:36.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e1ae251f-408a-4001-9aa9-ee4043234ba2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T13:59:36.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T13:59:36.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T13:59:36.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T13:59:36.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-09-13T13:59:36.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-09-13T13:59:36.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T13:59:36.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T13:59:36.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-09-13T13:59:36.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T13:59:36.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T13:59:36.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T13:59:36.429 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T13:59:36.430 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T13:59:36.430 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21519 2024-09-13T13:59:36.430 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21519/$cluster-$name.asok' 2024-09-13T13:59:36.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21519/$cluster-$name.asok' 2024-09-13T13:59:36.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T13:59:36.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-09-13T13:59:36.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-09-13T13:59:36.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T13:59:36.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T13:59:36.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T13:59:36.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T13:59:36.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T13:59:36.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T13:59:36.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/0 2024-09-13T13:59:36.431 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T13:59:36.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T13:59:36.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e1ae251f-408a-4001-9aa9-ee4043234ba2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.21519/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T13:59:36.432 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/0/whoami 2024-09-13T13:59:36.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T13:59:36.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T13:59:36.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T13:59:36.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T13:59:36.485 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:36.482+0000 7f00cf5057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:59:36.487 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:36.482+0000 7f00cf5057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:59:36.489 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:36.486+0000 7f00cf5057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:59:36.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T13:59:36.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T13:59:36.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T13:59:36.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T13:59:36.780 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T13:59:36.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T13:59:36.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T13:59:36.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T13:59:36.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T13:59:36.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T13:59:37.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T13:59:37.771 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:37.766+0000 7f00cf5057c0 -1 Falling back to public interface 2024-09-13T13:59:38.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T13:59:38.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T13:59:38.117 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T13:59:38.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T13:59:38.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T13:59:38.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T13:59:38.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T13:59:39.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T13:59:39.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T13:59:39.457 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T13:59:39.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T13:59:39.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T13:59:39.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T13:59:39.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T13:59:40.424 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:40.418+0000 7f00cf5057c0 -1 osd.0 30 log_to_monitors true 2024-09-13T13:59:40.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T13:59:40.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T13:59:40.787 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T13:59:40.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T13:59:40.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T13:59:40.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T13:59:41.183 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 33 up_thru 17 down_at 31 last_clean_interval [5,30) [v2:127.0.0.1:6800/302054358,v1:127.0.0.1:6801/302054358] [v2:127.0.0.1:6802/302054358,v1:127.0.0.1:6803/302054358] exists,up 0b3f1c54-7083-45c9-9be6-5d59b7dc95d3 2024-09-13T13:59:41.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T13:59:41.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T13:59:41.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T13:59:41.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-13T13:59:41.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=e1ae251f-408a-4001-9aa9-ee4043234ba2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T13:59:41.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-09-13T13:59:41.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2024-09-13T13:59:41.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T13:59:41.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T13:59:41.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T13:59:41.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/1 2024-09-13T13:59:41.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e1ae251f-408a-4001-9aa9-ee4043234ba2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T13:59:41.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T13:59:41.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T13:59:41.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T13:59:41.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-09-13T13:59:41.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-09-13T13:59:41.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T13:59:41.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T13:59:41.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-09-13T13:59:41.189 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T13:59:41.189 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T13:59:41.189 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T13:59:41.189 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T13:59:41.190 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T13:59:41.190 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21519 2024-09-13T13:59:41.190 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21519/$cluster-$name.asok' 2024-09-13T13:59:41.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21519/$cluster-$name.asok' 2024-09-13T13:59:41.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T13:59:41.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-09-13T13:59:41.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-09-13T13:59:41.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T13:59:41.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T13:59:41.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T13:59:41.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T13:59:41.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T13:59:41.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T13:59:41.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/1 2024-09-13T13:59:41.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T13:59:41.192 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T13:59:41.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e1ae251f-408a-4001-9aa9-ee4043234ba2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.21519/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T13:59:41.193 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/1/whoami 2024-09-13T13:59:41.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T13:59:41.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T13:59:41.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T13:59:41.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T13:59:41.229 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:41.226+0000 7fc66b9a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:59:41.266 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:41.262+0000 7fc66b9a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:59:41.267 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:41.262+0000 7fc66b9a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:59:41.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T13:59:41.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T13:59:41.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T13:59:41.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T13:59:41.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T13:59:41.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T13:59:41.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T13:59:41.573 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T13:59:41.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T13:59:41.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T13:59:41.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T13:59:42.347 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:42.342+0000 7fc66b9a37c0 -1 Falling back to public interface 2024-09-13T13:59:42.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T13:59:42.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T13:59:42.919 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T13:59:42.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T13:59:42.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T13:59:42.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T13:59:43.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T13:59:44.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T13:59:44.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T13:59:44.272 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T13:59:44.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T13:59:44.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T13:59:44.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T13:59:44.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T13:59:44.995 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:44.990+0000 7fc66b9a37c0 -1 osd.1 30 log_to_monitors true 2024-09-13T13:59:45.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T13:59:45.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T13:59:45.617 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T13:59:45.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T13:59:45.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T13:59:45.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T13:59:45.973 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 36 up_thru 36 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/340141500,v1:127.0.0.1:6811/340141500] [v2:127.0.0.1:6812/340141500,v1:127.0.0.1:6813/340141500] exists,up d4e5e593-574b-4dbd-a048-ee4e8d283396 2024-09-13T13:59:45.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T13:59:45.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T13:59:45.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T13:59:45.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-13T13:59:45.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=e1ae251f-408a-4001-9aa9-ee4043234ba2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T13:59:45.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-09-13T13:59:45.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2024-09-13T13:59:45.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T13:59:45.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T13:59:45.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T13:59:45.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/2 2024-09-13T13:59:45.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e1ae251f-408a-4001-9aa9-ee4043234ba2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T13:59:45.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T13:59:45.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T13:59:45.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T13:59:45.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-09-13T13:59:45.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-09-13T13:59:45.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T13:59:45.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T13:59:45.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-09-13T13:59:45.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T13:59:45.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T13:59:45.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T13:59:45.979 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T13:59:45.979 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T13:59:45.979 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21519 2024-09-13T13:59:45.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21519/$cluster-$name.asok' 2024-09-13T13:59:45.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21519/$cluster-$name.asok' 2024-09-13T13:59:45.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T13:59:45.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-09-13T13:59:45.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-09-13T13:59:45.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T13:59:45.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T13:59:45.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T13:59:45.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T13:59:45.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T13:59:45.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T13:59:45.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/2 2024-09-13T13:59:45.980 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T13:59:45.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T13:59:45.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e1ae251f-408a-4001-9aa9-ee4043234ba2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.21519/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T13:59:45.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/2/whoami 2024-09-13T13:59:45.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T13:59:45.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T13:59:45.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T13:59:45.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T13:59:46.020 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:46.018+0000 7f2a8ed317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:59:46.034 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:46.030+0000 7f2a8ed317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:59:46.036 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:46.034+0000 7f2a8ed317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T13:59:46.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T13:59:46.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T13:59:46.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T13:59:46.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T13:59:46.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T13:59:46.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T13:59:46.319 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T13:59:46.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T13:59:46.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T13:59:46.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T13:59:46.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T13:59:47.306 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:47.302+0000 7f2a8ed317c0 -1 Falling back to public interface 2024-09-13T13:59:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T13:59:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T13:59:47.746 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T13:59:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T13:59:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T13:59:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T13:59:48.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T13:59:49.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T13:59:49.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T13:59:49.086 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T13:59:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T13:59:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T13:59:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T13:59:49.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T13:59:49.974 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T13:59:49.970+0000 7f2a8ed317c0 -1 osd.2 30 log_to_monitors true 2024-09-13T13:59:50.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T13:59:50.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T13:59:50.411 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T13:59:50.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T13:59:50.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T13:59:50.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T13:59:51.019 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 40 up_thru 40 down_at 31 last_clean_interval [15,30) [v2:127.0.0.1:6818/2156113142,v1:127.0.0.1:6819/2156113142] [v2:127.0.0.1:6820/2156113142,v1:127.0.0.1:6821/2156113142] exists,up 4657d173-9a8d-4d21-aeba-8aad33c6cb4a 2024-09-13T13:59:51.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T13:59:51.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T13:59:51.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T13:59:51.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-09-13T13:59:52.023 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-09-13T13:59:52.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-09-13T13:59:52.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-13T13:59:52.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-09-13T13:59:52.165 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T13:59:52.165 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "19.3.0-4697-gd58f7306", 2024-09-13T13:59:52.166 INFO:tasks.workunit.client.0.smithi112.stdout: "release": "squid", 2024-09-13T13:59:52.166 INFO:tasks.workunit.client.0.smithi112.stdout: "release_type": "dev" 2024-09-13T13:59:52.166 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T13:59:52.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-13T13:59:52.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-09-13T13:59:52.306 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T13:59:52.306 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "19.3.0-4697-gd58f7306", 2024-09-13T13:59:52.306 INFO:tasks.workunit.client.0.smithi112.stdout: "release": "squid", 2024-09-13T13:59:52.306 INFO:tasks.workunit.client.0.smithi112.stdout: "release_type": "dev" 2024-09-13T13:59:52.306 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T13:59:52.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-13T13:59:52.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-09-13T13:59:52.474 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T13:59:52.474 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "19.3.0-4697-gd58f7306", 2024-09-13T13:59:52.474 INFO:tasks.workunit.client.0.smithi112.stdout: "release": "squid", 2024-09-13T13:59:52.474 INFO:tasks.workunit.client.0.smithi112.stdout: "release_type": "dev" 2024-09-13T13:59:52.474 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T13:59:52.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-09-13T13:59:52.528 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T13:59:52.528 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "objxxx", 2024-09-13T13:59:52.528 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T13:59:52.528 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:52.528 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 2, 2024-09-13T13:59:52.528 INFO:tasks.workunit.client.0.smithi112.stdout: "snapshots": [ 2024-09-13T13:59:52.528 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:52.528 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 1, 2024-09-13T13:59:52.528 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "snap01" 2024-09-13T13:59:52.529 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T13:59:52.529 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:52.529 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 2, 2024-09-13T13:59:52.529 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "snap02" 2024-09-13T13:59:52.529 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T13:59:52.529 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T13:59:52.529 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 5, 2024-09-13T13:59:52.529 INFO:tasks.workunit.client.0.smithi112.stdout: "overlaps": [] 2024-09-13T13:59:52.529 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T13:59:52.529 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:52.529 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 3, 2024-09-13T13:59:52.529 INFO:tasks.workunit.client.0.smithi112.stdout: "snapshots": [ 2024-09-13T13:59:52.529 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:52.529 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 3, 2024-09-13T13:59:52.529 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "snap13" 2024-09-13T13:59:52.530 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T13:59:52.530 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T13:59:52.530 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 6, 2024-09-13T13:59:52.530 INFO:tasks.workunit.client.0.smithi112.stdout: "overlaps": [] 2024-09-13T13:59:52.530 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T13:59:52.530 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:52.530 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 5, 2024-09-13T13:59:52.530 INFO:tasks.workunit.client.0.smithi112.stdout: "snapshots": [ 2024-09-13T13:59:52.530 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:52.530 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 4, 2024-09-13T13:59:52.530 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "snap24" 2024-09-13T13:59:52.530 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T13:59:52.530 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:52.530 INFO:tasks.workunit.client.0.smithi112.stdout: "id": 5, 2024-09-13T13:59:52.531 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "snap25" 2024-09-13T13:59:52.531 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T13:59:52.531 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T13:59:52.531 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 6, 2024-09-13T13:59:52.531 INFO:tasks.workunit.client.0.smithi112.stdout: "overlaps": [] 2024-09-13T13:59:52.531 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T13:59:52.531 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T13:59:52.531 INFO:tasks.workunit.client.0.smithi112.stdout: "id": "head", 2024-09-13T13:59:52.531 INFO:tasks.workunit.client.0.smithi112.stdout: "snapshots": [], 2024-09-13T13:59:52.531 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 6 2024-09-13T13:59:52.531 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T13:59:52.531 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T13:59:52.531 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T13:59:52.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-09-13T13:59:52.920 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is unset 2024-09-13T13:59:52.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-09-13T13:59:53.472 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is unset 2024-09-13T13:59:53.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-09-13T13:59:53.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-09-13T13:59:53.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-09-13T13:59:53.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-13T13:59:54.112 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T13:59:54.112 INFO:tasks.workunit.client.0.smithi112.stdout:1.3 4 0 4 0 0 23 0 0 7 0 7 active+undersized+degraded 2024-09-13T13:59:46.538106+0000 28'7 38:56 [1,0] 1 [1,0] 1 28'7 2024-09-13T13:59:01.893042+0000 28'7 2024-09-13T13:59:01.893042+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:59:01.893042+0000 4 0 2024-09-13T13:59:54.112 INFO:tasks.workunit.client.0.smithi112.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 peering 2024-09-13T13:59:50.454266+0000 0'0 39:33 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 no scrub is scheduled 0 0 2024-09-13T13:59:54.112 INFO:tasks.workunit.client.0.smithi112.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 undersized+peered 2024-09-13T13:59:41.416437+0000 0'0 35:11 [0] 0 [0] 0 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T13:59:54.113 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-13T13:59:46.131779+0000 0'0 38:22 [1,0] 1 [1,0] 1 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T13:59:54.113 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T13:59:54.113 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T13:59:54.113 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T13:59:54.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-09-13T13:59:56.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-09-13T13:59:56.268 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T13:59:56.268 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T13:59:56.269 INFO:tasks.workunit.client.0.smithi112.stdout: "must": true, 2024-09-13T13:59:56.269 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "0.000000" 2024-09-13T13:59:56.269 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T13:59:56.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-09-13T14:00:01.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-13T14:00:01.584 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T14:00:01.584 INFO:tasks.workunit.client.0.smithi112.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-09-13T13:59:51.528224+0000 28'7 44:74 [1,2,0] 1 [1,2,0] 1 28'7 2024-09-13T13:59:01.893042+0000 28'7 2024-09-13T13:59:01.893042+0000 0 1 periodic scrub scheduled @ 2024-09-14T13:59:01.893042+0000 4 0 2024-09-13T14:00:01.584 INFO:tasks.workunit.client.0.smithi112.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T13:59:51.083923+0000 0'0 44:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T14:00:01.584 INFO:tasks.workunit.client.0.smithi112.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T13:59:51.159242+0000 0'0 44:29 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T14:00:01.584 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T13:59:51.436028+0000 0'0 44:39 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T14:00:01.584 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T14:00:01.584 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T14:00:01.584 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:00:01.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-13T14:00:01.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-13T14:00:01.601 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T13:59:57.210+0000 7fc64e12a640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-09-13T14:00:01.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-09-13T14:00:01.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-09-13T14:00:01.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-13T14:00:01.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-09-13T14:00:01.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-09-13T14:00:01.608 INFO:tasks.workunit.client.0.smithi112.stdout:prev count: 1 2024-09-13T14:00:01.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-09-13T14:00:01.733 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T14:00:01.734 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T14:00:01.734 INFO:tasks.workunit.client.0.smithi112.stdout: "must": true, 2024-09-13T14:00:01.734 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "0.000000" 2024-09-13T14:00:01.734 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:00:01.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-09-13T14:00:06.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-13T14:00:07.047 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T14:00:07.047 INFO:tasks.workunit.client.0.smithi112.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-09-13T14:00:02.324922+0000 28'7 44:90 [1,2,0] 1 [1,2,0] 1 28'7 2024-09-13T14:00:02.324779+0000 28'7 2024-09-13T14:00:02.324779+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:00:02.324779+0000 4 0 2024-09-13T14:00:07.047 INFO:tasks.workunit.client.0.smithi112.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T13:59:51.083923+0000 0'0 44:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T14:00:07.047 INFO:tasks.workunit.client.0.smithi112.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T13:59:51.159242+0000 0'0 44:29 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T14:00:07.047 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T13:59:51.436028+0000 0'0 44:39 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T13:58:50.868117+0000 0'0 2024-09-13T13:58:50.868117+0000 0 0 periodic scrub scheduled @ 2024-09-14T13:58:50.868117+0000 0 0 2024-09-13T14:00:07.047 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T14:00:07.047 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T14:00:07.048 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:00:07.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-13T14:00:07.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-13T14:00:07.065 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T13:59:57.210+0000 7fc64e12a640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-09-13T14:00:07.065 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-13T14:00:07.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-09-13T14:00:07.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-09-13T14:00:07.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-13T14:00:07.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-09-13T14:00:07.068 INFO:tasks.workunit.client.0.smithi112.stdout:current count: 1 2024-09-13T14:00:07.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-09-13T14:00:07.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-09-13T14:00:07.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:00:07.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:00:07.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:00:07.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:00:07.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:00:08.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:00:08.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-09-13T14:00:08.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-13T14:00:08.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-13T14:00:08.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-09-13T14:00:08.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-13T14:00:09.482 INFO:tasks.workunit.client.0.smithi112.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T14:00:09.482 INFO:tasks.workunit.client.0.smithi112.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-13T14:00:09.482 INFO:tasks.workunit.client.0.smithi112.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-13T14:00:09.482 INFO:tasks.workunit.client.0.smithi112.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-13T14:00:09.482 INFO:tasks.workunit.client.0.smithi112.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T14:00:09.482 INFO:tasks.workunit.client.0.smithi112.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-13T14:00:09.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-13T14:00:09.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-09-13T14:00:09.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-13T14:00:09.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-09-13T14:00:10.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-09-13T14:00:10.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-09-13T14:00:10.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-09-13T14:00:10.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-09-13T14:00:10.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2024-09-13T14:00:10.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:00:10.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2024-09-13T14:00:10.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:00:10.581 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:00:10.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:00:10.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:00:10.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:00:10.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:00:10.690 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:00:10.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:00:10.691 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:00:10.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:00:10.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:00:10.693 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:00:10.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:00:10.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:00:10.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:00:10.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:00:10.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:00:10.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:00:10.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2024-09-13T14:00:10.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:00:10.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:00:10.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21519 2024-09-13T14:00:10.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21519 2024-09-13T14:00:10.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:00:10.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:00:10.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-13T14:00:10.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-mapper 0 2024-09-13T14:00:10.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2024-09-13T14:00:10.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-13T14:00:10.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2024-09-13T14:00:10.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:00:10.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:00:10.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:00:10.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:00:10.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:00:10.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:00:10.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:00:10.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:00:10.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:00:10.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:00:10.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:00:10.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:00:10.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:00:10.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:00:10.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:00:10.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:00:10.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:00:10.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-13T14:00:10.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2024-09-13T14:00:10.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:00:10.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:00:10.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21519 2024-09-13T14:00:10.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21519 2024-09-13T14:00:10.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:00:10.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:00:10.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-13T14:00:10.727 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T14:00:10.728 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T14:00:10.777 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-09-13T14:00:10.777 DEBUG:teuthology.orchestra.run.smithi112:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d58f73069a8157a9ace12a0247ee3e8497707ea9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-09-13T14:00:10.829 INFO:tasks.workunit.client.0.smithi112.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-13T14:00:10.835 INFO:tasks.workunit.client.0.smithi112.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-13T14:00:10.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-13T14:00:10.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-13T14:00:10.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-13T14:00:10.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-13T14:00:10.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-13T14:00:10.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-13T14:00:10.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-13T14:00:10.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-13T14:00:10.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-recovery-scrub 2024-09-13T14:00:10.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-09-13T14:00:10.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-09-13T14:00:10.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-09-13T14:00:10.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-09-13T14:00:10.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-09-13T14:00:10.837 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-09-13T14:00:10.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none ' 2024-09-13T14:00:10.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-09-13T14:00:10.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-09-13T14:00:10.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-13T14:00:10.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-09-13T14:00:10.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-13T14:00:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-09-13T14:00:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_recovery_scrub_2' 2024-09-13T14:00:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-09-13T14:00:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-09-13T14:00:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-09-13T14:00:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-09-13T14:00:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.37913 2024-09-13T14:00:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-09-13T14:00:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-09-13T14:00:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-09-13T14:00:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-09-13T14:00:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-09-13T14:00:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-recovery-scrub 2024-09-13T14:00:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-recovery-scrub 2024-09-13T14:00:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2024-09-13T14:00:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:00:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-13T14:00:10.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:00:10.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:00:10.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:00:10.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:00:10.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:00:10.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:00:10.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:00:10.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:00:10.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:00:10.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:00:10.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:00:10.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:00:10.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:00:10.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:00:10.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:00:10.846 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:00:10.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:00:10.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:00:10.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2024-09-13T14:00:10.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:00:10.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:00:10.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:00:10.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37913 2024-09-13T14:00:10.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:00:10.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:00:10.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-recovery-scrub 2024-09-13T14:00:10.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:00:10.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:00:10.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:00:10.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.37913 2024-09-13T14:00:10.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:00:10.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-13T14:00:10.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-13T14:00:10.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:00:10.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-09-13T14:00:10.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-13T14:00:10.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-recovery-scrub 2024-09-13T14:00:10.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T14:00:10.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T14:00:10.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T14:00:10.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-recovery-scrub/a 2024-09-13T14:00:10.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-13T14:00:11.313 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T14:00:11.313 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:00:11.313 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:00:11.313 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:00:11.313 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:00:11.313 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:00:11.314 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:00:11.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-13T14:00:11.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T14:00:11.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T14:00:11.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:00:11.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:00:11.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T14:00:11.533 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T14:00:11.533 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:00:11.533 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:00:11.533 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:00:11.533 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:00:11.533 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:00:11.533 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:00:11.533 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37913/ceph-mon.a.asok 2024-09-13T14:00:11.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:00:11.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.37913/ceph-mon.a.asok config get fsid 2024-09-13T14:00:11.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T14:00:11.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:00:11.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:00:11.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T14:00:11.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T14:00:11.645 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:00:11.645 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:00:11.645 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:00:11.645 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:00:11.645 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:00:11.645 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:00:11.645 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37913/ceph-mon.a.asok 2024-09-13T14:00:11.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:00:11.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.37913/ceph-mon.a.asok config get mon_host 2024-09-13T14:00:11.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-09-13T14:00:11.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-recovery-scrub 2024-09-13T14:00:11.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T14:00:11.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T14:00:11.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T14:00:11.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-recovery-scrub/x 2024-09-13T14:00:11.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T14:00:12.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T14:00:12.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:00:12.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:00:12.606 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:00:12.606 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:00:12.606 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:00:12.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:00:12.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:00:12.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:00:12.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-09-13T14:00:12.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-09-13T14:00:12.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-09-13T14:00:12.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-09-13T14:00:12.644 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-09-13T14:00:12.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-09-13T14:00:12.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:00:12.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-09-13T14:00:12.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-13T14:00:12.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:00:12.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T14:00:12.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:00:12.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-09-13T14:00:12.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-13T14:00:12.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:00:12.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:00:12.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:00:12.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-09-13T14:00:12.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-09-13T14:00:12.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:00:12.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:00:12.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-13T14:00:12.649 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:00:12.650 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:00:12.650 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:00:12.650 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:00:12.650 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:00:12.650 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:00:12.650 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:00:12.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:00:12.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:00:12.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:00:12.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:00:12.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-13T14:00:12.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-13T14:00:12.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:00:12.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:00:12.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:00:12.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:00:12.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:00:12.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-13T14:00:12.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-09-13T14:00:12.651 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:00:12.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=527c7dd0-a5af-4c8e-9fa0-f1aaeffd0916 2024-09-13T14:00:12.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 527c7dd0-a5af-4c8e-9fa0-f1aaeffd0916' 2024-09-13T14:00:12.652 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 527c7dd0-a5af-4c8e-9fa0-f1aaeffd0916 2024-09-13T14:00:12.652 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:00:12.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBsReRmSFGuJxAAh8QAfu3NXazKjS7Pcjo4SQ== 2024-09-13T14:00:12.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBsReRmSFGuJxAAh8QAfu3NXazKjS7Pcjo4SQ=="}' 2024-09-13T14:00:12.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 527c7dd0-a5af-4c8e-9fa0-f1aaeffd0916 -i td/osd-recovery-scrub/0/new.json 2024-09-13T14:00:12.893 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:00:12.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-09-13T14:00:12.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBsReRmSFGuJxAAh8QAfu3NXazKjS7Pcjo4SQ== --osd-uuid 527c7dd0-a5af-4c8e-9fa0-f1aaeffd0916 2024-09-13T14:00:12.950 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:12.945+0000 7ff5bf4af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:12.952 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:12.949+0000 7ff5bf4af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:12.954 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:12.949+0000 7ff5bf4af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:12.954 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:12.949+0000 7ff5bf4af7c0 -1 bdev(0x556453051c00 td/osd-recovery-scrub/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:00:12.954 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:12.949+0000 7ff5bf4af7c0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-09-13T14:00:19.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-09-13T14:00:19.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:00:19.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T14:00:19.084 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:00:19.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:00:19.494 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:00:19.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T14:00:19.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-13T14:00:19.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:00:19.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:00:19.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:00:19.528 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:19.521+0000 7f7a26bd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:19.530 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:19.525+0000 7f7a26bd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:19.531 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:19.525+0000 7f7a26bd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:19.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T14:00:19.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:00:19.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:00:19.856 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:00:19.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:00:19.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:00:19.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:19.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:00:19.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:19.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:00:20.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:00:20.811 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:20.805+0000 7f7a26bd47c0 -1 Falling back to public interface 2024-09-13T14:00:21.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:00:21.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:21.197 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:00:21.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:00:21.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:21.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:00:21.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:00:22.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:00:22.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:22.518 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:00:22.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:00:22.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:22.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:00:22.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:00:23.453 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:23.449+0000 7f7a26bd47c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:00:23.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:00:23.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:23.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:00:23.841 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:00:23.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:23.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:00:24.204 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2595341913,v1:127.0.0.1:6803/2595341913] [v2:127.0.0.1:6804/2595341913,v1:127.0.0.1:6805/2595341913] exists,up 527c7dd0-a5af-4c8e-9fa0-f1aaeffd0916 2024-09-13T14:00:24.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:00:24.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:00:24.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:00:24.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:00:24.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-09-13T14:00:24.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-13T14:00:24.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:00:24.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T14:00:24.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:00:24.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-09-13T14:00:24.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-13T14:00:24.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:00:24.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:00:24.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:00:24.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-09-13T14:00:24.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-09-13T14:00:24.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:00:24.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:00:24.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-13T14:00:24.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:00:24.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:00:24.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:00:24.209 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:00:24.210 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:00:24.210 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:00:24.210 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:00:24.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:00:24.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:00:24.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:00:24.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:00:24.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-13T14:00:24.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-13T14:00:24.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:00:24.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:00:24.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:00:24.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:00:24.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:00:24.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-13T14:00:24.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-09-13T14:00:24.211 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:00:24.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=90cadcfd-8374-44e7-b9df-d68d4c8c5a89 2024-09-13T14:00:24.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 90cadcfd-8374-44e7-b9df-d68d4c8c5a89' 2024-09-13T14:00:24.211 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 90cadcfd-8374-44e7-b9df-d68d4c8c5a89 2024-09-13T14:00:24.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:00:24.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB4ReRmT1S1DRAAV9q4HkM8mPZWQ4T9NHAlxw== 2024-09-13T14:00:24.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB4ReRmT1S1DRAAV9q4HkM8mPZWQ4T9NHAlxw=="}' 2024-09-13T14:00:24.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 90cadcfd-8374-44e7-b9df-d68d4c8c5a89 -i td/osd-recovery-scrub/1/new.json 2024-09-13T14:00:24.719 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:00:24.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-09-13T14:00:24.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQB4ReRmT1S1DRAAV9q4HkM8mPZWQ4T9NHAlxw== --osd-uuid 90cadcfd-8374-44e7-b9df-d68d4c8c5a89 2024-09-13T14:00:24.768 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:24.761+0000 7f81cd6547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:24.798 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:24.793+0000 7f81cd6547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:24.799 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:24.793+0000 7f81cd6547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:24.800 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:24.797+0000 7f81cd6547c0 -1 bdev(0x561c57e51c00 td/osd-recovery-scrub/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:00:24.800 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:24.797+0000 7f81cd6547c0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-09-13T14:00:30.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-09-13T14:00:30.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:00:30.931 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:00:30.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T14:00:30.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:00:31.339 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:00:31.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T14:00:31.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:00:31.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-13T14:00:31.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:00:31.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:00:31.373 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:31.369+0000 7f14e1a347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:31.375 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:31.369+0000 7f14e1a347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:31.376 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:31.373+0000 7f14e1a347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:31.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T14:00:31.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:00:31.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:00:31.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:00:31.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:00:31.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:31.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:00:31.699 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:00:31.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:31.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:00:32.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:00:32.922 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:32.917+0000 7f14e1a347c0 -1 Falling back to public interface 2024-09-13T14:00:33.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:00:33.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:33.017 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:00:33.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:00:33.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:33.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:00:33.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:00:34.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:00:34.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:34.352 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:00:34.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:00:34.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:34.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:00:34.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:00:35.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:00:35.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:35.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:00:35.674 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:00:35.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:35.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:00:35.938 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:35.933+0000 7f14e1a347c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:00:36.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:00:37.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:00:37.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:37.151 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:00:37.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:00:37.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:37.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:00:37.485 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2493075221,v1:127.0.0.1:6811/2493075221] [v2:127.0.0.1:6812/2493075221,v1:127.0.0.1:6813/2493075221] exists,up 90cadcfd-8374-44e7-b9df-d68d4c8c5a89 2024-09-13T14:00:37.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:00:37.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:00:37.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:00:37.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:00:37.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-09-13T14:00:37.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-13T14:00:37.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:00:37.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T14:00:37.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:00:37.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-09-13T14:00:37.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-13T14:00:37.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:00:37.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:00:37.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:00:37.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-09-13T14:00:37.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-09-13T14:00:37.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:00:37.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:00:37.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-13T14:00:37.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:00:37.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:00:37.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:00:37.490 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:00:37.490 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:00:37.490 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:00:37.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:00:37.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:00:37.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:00:37.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:00:37.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:00:37.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-13T14:00:37.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-13T14:00:37.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:00:37.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:00:37.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:00:37.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:00:37.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:00:37.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-13T14:00:37.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-09-13T14:00:37.494 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:00:37.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bb3b13af-bd25-4e69-a217-e8ca2a1fa559 2024-09-13T14:00:37.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 bb3b13af-bd25-4e69-a217-e8ca2a1fa559' 2024-09-13T14:00:37.494 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 bb3b13af-bd25-4e69-a217-e8ca2a1fa559 2024-09-13T14:00:37.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:00:37.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCFReRmeH6WHhAA+R810Nl1rlTc7tgmBnFvxg== 2024-09-13T14:00:37.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCFReRmeH6WHhAA+R810Nl1rlTc7tgmBnFvxg=="}' 2024-09-13T14:00:37.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bb3b13af-bd25-4e69-a217-e8ca2a1fa559 -i td/osd-recovery-scrub/2/new.json 2024-09-13T14:00:37.894 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:00:37.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-09-13T14:00:37.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCFReRmeH6WHhAA+R810Nl1rlTc7tgmBnFvxg== --osd-uuid bb3b13af-bd25-4e69-a217-e8ca2a1fa559 2024-09-13T14:00:37.951 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:37.945+0000 7f286ea037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:37.980 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:37.977+0000 7f286ea037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:37.981 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:37.977+0000 7f286ea037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:37.982 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:37.977+0000 7f286ea037c0 -1 bdev(0x5618a2567c00 td/osd-recovery-scrub/2/block) open stat got: (1) Operation not permitted 2024-09-13T14:00:37.982 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:37.977+0000 7f286ea037c0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-09-13T14:00:43.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-09-13T14:00:43.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:00:43.782 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T14:00:43.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T14:00:43.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:00:44.203 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T14:00:44.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T14:00:44.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:00:44.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:00:44.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-13T14:00:44.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:00:44.237 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:44.232+0000 7fcb3a1627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:44.266 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:44.260+0000 7fcb3a1627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:44.267 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:44.264+0000 7fcb3a1627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:44.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T14:00:44.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:00:44.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:00:44.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:00:44.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:00:44.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:44.549 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:00:44.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:00:44.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:44.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:00:44.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:00:45.558 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:45.552+0000 7fcb3a1627c0 -1 Falling back to public interface 2024-09-13T14:00:45.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:00:45.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:45.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:00:45.882 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:00:45.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:00:45.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:46.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:00:47.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:00:47.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:47.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:00:47.201 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:00:47.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:47.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:00:47.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:00:48.479 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:48.472+0000 7fcb3a1627c0 -1 osd.2 0 log_to_monitors true 2024-09-13T14:00:48.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:00:48.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:48.527 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:00:48.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:00:48.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:48.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:00:48.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:00:49.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:00:49.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:49.902 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:00:49.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:00:49.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:49.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:00:50.222 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2957001664,v1:127.0.0.1:6819/2957001664] [v2:127.0.0.1:6820/2957001664,v1:127.0.0.1:6821/2957001664] exists,up bb3b13af-bd25-4e69-a217-e8ca2a1fa559 2024-09-13T14:00:50.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:00:50.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:00:50.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:00:50.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:00:50.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-09-13T14:00:50.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-13T14:00:50.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:00:50.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-13T14:00:50.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:00:50.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-09-13T14:00:50.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-13T14:00:50.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:00:50.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:00:50.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:00:50.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-09-13T14:00:50.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-09-13T14:00:50.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:00:50.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:00:50.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-13T14:00:50.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:00:50.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:00:50.228 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:00:50.228 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:00:50.228 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:00:50.228 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:00:50.228 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:00:50.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:00:50.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:00:50.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:00:50.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:00:50.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-13T14:00:50.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-13T14:00:50.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:00:50.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:00:50.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:00:50.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:00:50.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:00:50.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-13T14:00:50.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-09-13T14:00:50.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:00:50.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=76504669-1c06-490e-bcdc-5e6880ab5b51 2024-09-13T14:00:50.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 76504669-1c06-490e-bcdc-5e6880ab5b51' 2024-09-13T14:00:50.230 INFO:tasks.workunit.client.0.smithi112.stdout:add osd3 76504669-1c06-490e-bcdc-5e6880ab5b51 2024-09-13T14:00:50.230 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:00:50.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCSReRmFgeZDhAATZnRw85RO6hQgOcrBCjBsg== 2024-09-13T14:00:50.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCSReRmFgeZDhAATZnRw85RO6hQgOcrBCjBsg=="}' 2024-09-13T14:00:50.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 76504669-1c06-490e-bcdc-5e6880ab5b51 -i td/osd-recovery-scrub/3/new.json 2024-09-13T14:00:50.605 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:00:50.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-09-13T14:00:50.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCSReRmFgeZDhAATZnRw85RO6hQgOcrBCjBsg== --osd-uuid 76504669-1c06-490e-bcdc-5e6880ab5b51 2024-09-13T14:00:50.656 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:50.652+0000 7fae1dee37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:50.689 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:50.684+0000 7fae1dee37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:50.691 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:50.684+0000 7fae1dee37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:50.691 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:50.684+0000 7fae1dee37c0 -1 bdev(0x55c661e69c00 td/osd-recovery-scrub/3/block) open stat got: (1) Operation not permitted 2024-09-13T14:00:50.691 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:50.688+0000 7fae1dee37c0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-09-13T14:00:56.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-09-13T14:00:56.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:00:56.835 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd3 key to auth repository 2024-09-13T14:00:56.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-13T14:00:56.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:00:57.247 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.3 2024-09-13T14:00:57.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-13T14:00:57.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-13T14:00:57.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:00:57.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:00:57.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:00:57.280 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:57.276+0000 7fb2c7d457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:57.283 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:57.276+0000 7fb2c7d457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:57.283 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:57.280+0000 7fb2c7d457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:00:57.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-13T14:00:57.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:00:57.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:00:57.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:00:57.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:00:57.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:57.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:00:57.615 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:00:57.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:57.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:00:57.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:00:58.346 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:00:58.344+0000 7fb2c7d457c0 -1 Falling back to public interface 2024-09-13T14:00:58.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:00:58.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:00:58.960 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:00:58.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:00:58.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:00:58.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:00:59.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:01:00.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:01:00.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:01:00.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:01:00.282 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:01:00.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:01:00.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:01:00.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:01:01.102 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:01:01.100+0000 7fb2c7d457c0 -1 osd.3 0 log_to_monitors true 2024-09-13T14:01:01.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:01:01.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:01:01.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:01:01.608 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:01:01.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:01:01.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:01:01.948 INFO:tasks.workunit.client.0.smithi112.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/405939934,v1:127.0.0.1:6827/405939934] [v2:127.0.0.1:6828/405939934,v1:127.0.0.1:6829/405939934] exists,up 76504669-1c06-490e-bcdc-5e6880ab5b51 2024-09-13T14:01:01.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:01:01.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:01:01.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:01:01.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-09-13T14:01:01.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-13T14:01:02.474 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'test' created 2024-09-13T14:01:02.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:01:03.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-09-13T14:01:03.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:01:03.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:01:03.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:01:03.497 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:01:03.497 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:01:03.497 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:01:03.498 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:01:03.498 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:01:03.498 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:01:03.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:01:03.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:01:03.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:01:03.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:01:03.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:01:03.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:01:04.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:01:04.096 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:01:04.096 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:01:04.096 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T14:01:04.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:01:04.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:01:04.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:01:04.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836489 2024-09-13T14:01:04.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836489 2024-09-13T14:01:04.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489' 2024-09-13T14:01:04.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:01:04.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:01:04.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672968 2024-09-13T14:01:04.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672968 2024-09-13T14:01:04.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2024-09-13T14:01:04.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:01:04.382 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:01:04.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509444 2024-09-13T14:01:04.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509444 2024-09-13T14:01:04.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509444' 2024-09-13T14:01:04.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:01:04.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:01:04.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345922 2024-09-13T14:01:04.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345922 2024-09-13T14:01:04.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509444 3-85899345922' 2024-09-13T14:01:04.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:01:04.669 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836489 2024-09-13T14:01:04.669 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:01:04.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:01:04.671 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836489 2024-09-13T14:01:04.671 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:01:04.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836489 2024-09-13T14:01:04.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-09-13T14:01:04.672 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836489 2024-09-13T14:01:04.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:01:04.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836489 2024-09-13T14:01:04.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:01:05.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:01:05.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:01:06.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836489 2024-09-13T14:01:06.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:01:06.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672968 2024-09-13T14:01:06.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:01:06.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:01:06.302 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672968 2024-09-13T14:01:06.302 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:01:06.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672968 2024-09-13T14:01:06.303 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672968 2024-09-13T14:01:06.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-09-13T14:01:06.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:01:06.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672968 -lt 42949672968 2024-09-13T14:01:06.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:01:06.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509444 2024-09-13T14:01:06.626 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:01:06.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:01:06.628 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509444 2024-09-13T14:01:06.628 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:01:06.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509444 2024-09-13T14:01:06.630 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509444 2024-09-13T14:01:06.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-09-13T14:01:06.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:01:06.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509445 -lt 64424509444 2024-09-13T14:01:06.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:01:06.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345922 2024-09-13T14:01:06.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:01:06.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:01:06.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345922 2024-09-13T14:01:06.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:01:06.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345922 2024-09-13T14:01:06.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-09-13T14:01:06.971 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.3 seq 85899345922 2024-09-13T14:01:06.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:01:07.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345922 -lt 85899345922 2024-09-13T14:01:07.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:01:07.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:01:07.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:01:07.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:01:07.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:01:07.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:01:07.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:01:07.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:01:07.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:01:07.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:01:07.714 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:01:08.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:01:08.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:01:08.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:01:08.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:01:08.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:01:08.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:01:08.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:01:08.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-09-13T14:01:08.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-09-13T14:01:08.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-09-13T14:01:08.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-09-13T14:01:08.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-09-13T14:01:09.006 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T14:01:09.006 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T14:01:03.455064+0000 0'0 23:9 [1] 1 [1] 1 0'0 2024-09-13T14:01:02.373976+0000 0'0 2024-09-13T14:01:02.373976+0000 0 0 periodic scrub scheduled @ 2024-09-14T18:14:28.298626+0000 0 0 2024-09-13T14:01:09.006 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T14:01:09.006 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T14:01:09.006 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:01:09.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.37913 bs=1M count=50 2024-09-13T14:01:09.181 INFO:tasks.workunit.client.0.smithi112.stderr:50+0 records in 2024-09-13T14:01:09.182 INFO:tasks.workunit.client.0.smithi112.stderr:50+0 records out 2024-09-13T14:01:09.182 INFO:tasks.workunit.client.0.smithi112.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.158503 s, 331 MB/s 2024-09-13T14:01:09.183 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-09-13T14:01:09.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:09.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.37913 2024-09-13T14:01:10.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:10.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.37913 2024-09-13T14:01:12.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:12.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.37913 2024-09-13T14:01:13.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:13.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.37913 2024-09-13T14:01:15.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:15.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.37913 2024-09-13T14:01:16.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:16.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.37913 2024-09-13T14:01:18.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:18.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.37913 2024-09-13T14:01:19.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:19.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.37913 2024-09-13T14:01:21.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:21.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.37913 2024-09-13T14:01:22.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:22.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.37913 2024-09-13T14:01:24.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:24.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.37913 2024-09-13T14:01:25.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:25.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.37913 2024-09-13T14:01:27.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:27.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.37913 2024-09-13T14:01:28.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:28.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.37913 2024-09-13T14:01:30.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:30.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.37913 2024-09-13T14:01:31.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:31.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.37913 2024-09-13T14:01:33.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:33.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.37913 2024-09-13T14:01:35.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:35.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.37913 2024-09-13T14:01:36.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:36.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.37913 2024-09-13T14:01:38.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:38.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.37913 2024-09-13T14:01:39.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:39.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.37913 2024-09-13T14:01:41.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:41.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.37913 2024-09-13T14:01:42.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:42.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.37913 2024-09-13T14:01:44.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:44.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.37913 2024-09-13T14:01:46.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:46.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.37913 2024-09-13T14:01:48.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:48.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.37913 2024-09-13T14:01:49.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:49.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.37913 2024-09-13T14:01:51.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:51.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.37913 2024-09-13T14:01:52.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:52.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.37913 2024-09-13T14:01:54.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:54.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.37913 2024-09-13T14:01:56.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:56.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.37913 2024-09-13T14:01:57.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:57.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.37913 2024-09-13T14:01:59.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:01:59.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.37913 2024-09-13T14:02:00.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:00.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.37913 2024-09-13T14:02:02.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:02.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.37913 2024-09-13T14:02:04.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:04.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.37913 2024-09-13T14:02:05.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:05.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.37913 2024-09-13T14:02:07.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:07.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.37913 2024-09-13T14:02:08.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:08.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.37913 2024-09-13T14:02:10.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:10.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.37913 2024-09-13T14:02:11.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:11.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.37913 2024-09-13T14:02:13.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:13.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.37913 2024-09-13T14:02:14.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:14.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.37913 2024-09-13T14:02:16.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:16.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.37913 2024-09-13T14:02:17.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:17.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.37913 2024-09-13T14:02:19.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:19.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.37913 2024-09-13T14:02:20.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:20.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.37913 2024-09-13T14:02:22.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:22.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.37913 2024-09-13T14:02:24.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:24.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.37913 2024-09-13T14:02:25.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:25.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.37913 2024-09-13T14:02:27.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:27.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.37913 2024-09-13T14:02:28.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:28.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.37913 2024-09-13T14:02:30.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:30.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.37913 2024-09-13T14:02:31.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:31.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.37913 2024-09-13T14:02:33.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:33.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.37913 2024-09-13T14:02:34.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:34.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.37913 2024-09-13T14:02:36.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:36.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.37913 2024-09-13T14:02:38.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:38.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.37913 2024-09-13T14:02:39.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:39.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.37913 2024-09-13T14:02:41.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:41.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.37913 2024-09-13T14:02:42.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:42.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.37913 2024-09-13T14:02:44.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:44.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.37913 2024-09-13T14:02:45.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:45.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.37913 2024-09-13T14:02:47.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:47.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.37913 2024-09-13T14:02:48.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:48.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.37913 2024-09-13T14:02:50.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:50.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.37913 2024-09-13T14:02:52.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:52.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.37913 2024-09-13T14:02:53.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:53.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.37913 2024-09-13T14:02:55.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:55.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.37913 2024-09-13T14:02:56.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:56.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.37913 2024-09-13T14:02:58.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:58.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.37913 2024-09-13T14:02:59.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:02:59.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.37913 2024-09-13T14:03:01.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:01.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.37913 2024-09-13T14:03:02.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:02.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.37913 2024-09-13T14:03:04.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:04.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.37913 2024-09-13T14:03:05.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:05.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.37913 2024-09-13T14:03:07.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:07.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.37913 2024-09-13T14:03:08.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:08.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.37913 2024-09-13T14:03:10.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:10.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.37913 2024-09-13T14:03:11.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:11.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.37913 2024-09-13T14:03:13.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:13.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.37913 2024-09-13T14:03:14.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:14.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.37913 2024-09-13T14:03:16.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:16.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.37913 2024-09-13T14:03:17.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:17.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.37913 2024-09-13T14:03:19.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:19.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.37913 2024-09-13T14:03:20.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:20.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.37913 2024-09-13T14:03:22.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:22.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.37913 2024-09-13T14:03:23.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:23.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.37913 2024-09-13T14:03:25.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:25.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.37913 2024-09-13T14:03:26.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:26.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.37913 2024-09-13T14:03:28.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:28.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.37913 2024-09-13T14:03:29.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:29.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.37913 2024-09-13T14:03:31.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:31.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.37913 2024-09-13T14:03:32.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:32.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.37913 2024-09-13T14:03:34.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:34.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.37913 2024-09-13T14:03:35.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:35.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.37913 2024-09-13T14:03:37.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:37.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.37913 2024-09-13T14:03:38.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:38.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.37913 2024-09-13T14:03:40.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:40.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.37913 2024-09-13T14:03:42.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-13T14:03:42.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.37913 2024-09-13T14:03:43.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.37913 2024-09-13T14:03:43.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-09-13T14:03:43.876 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 1 size to 4 2024-09-13T14:03:43.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-09-13T14:03:43.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-09-13T14:03:43.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-13T14:03:43.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-13T14:03:43.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-13T14:03:43.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-13T14:03:44.230 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:03:44.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-09-13T14:03:46.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-09-13T14:03:46.251 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-09-13T14:03:46.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-09-13T14:03:46.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-13T14:03:46.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-13T14:03:46.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-13T14:03:46.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-13T14:03:46.574 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:03:46.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-09-13T14:03:48.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-09-13T14:03:48.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-09-13T14:03:48.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-09-13T14:03:48.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-13T14:03:48.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-13T14:03:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-13T14:03:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-13T14:03:48.913 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:03:48.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-09-13T14:03:50.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2024-09-13T14:03:50.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2024-09-13T14:03:50.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2024-09-13T14:03:50.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-13T14:03:50.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-13T14:03:50.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-13T14:03:50.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-13T14:03:51.254 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:03:51.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-09-13T14:03:53.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 3 -eq 10 2024-09-13T14:03:53.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 3 + 1 2024-09-13T14:03:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=4 2024-09-13T14:03:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-13T14:03:53.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-13T14:03:53.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-13T14:03:53.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-13T14:03:53.740 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:03:53.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-09-13T14:03:53.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-09-13T14:03:53.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-09-13T14:03:54.281 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T14:03:54.281 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 100 0 297 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-09-13T14:03:47.670583+0000 24'1300 28:1329 [1,0,2,3] 1 [1,0] 1 0'0 2024-09-13T14:01:02.373976+0000 0'0 2024-09-13T14:01:02.373976+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:03:54.281 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T14:03:54.281 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T14:03:54.281 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:03:54.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-09-13T14:04:04.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-09-13T14:04:04.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:04:04.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:04:04.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:04:04.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:04:04.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:04:09.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:04:09.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-09-13T14:04:09.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress.*scrubs' 2024-09-13T14:04:09.621 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-09-13T14:04:09.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-09-13T14:04:09.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:04:09.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2024-09-13T14:04:09.627 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:03:51.173+0000 7f7a1fb88640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.627 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:03:53.193+0000 7f7a1fb88640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.627 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:03:55.153+0000 7f7a1fb88640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.627 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:03:58.169+0000 7f7a1fb88640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.627 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:04:00.221+0000 7f7a1fb88640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.627 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:04:02.161+0000 7f7a1fb88640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:04:09.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2024-09-13T14:04:09.646 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:03:54.329+0000 7f14da9e8640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.646 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:03:55.349+0000 7f14da9e8640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.646 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:03:56.321+0000 7f14da9e8640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.646 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:03:58.241+0000 7f14da9e8640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.646 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:03:59.209+0000 7f14da9e8640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.646 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:04:01.269+0000 7f14da9e8640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.646 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:04:02.225+0000 7f14da9e8640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.646 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:04:03.245+0000 7f14da9e8640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.646 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:04:07.229+0000 7f14da9e8640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:04:09.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2024-09-13T14:04:09.650 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:03:50.501+0000 7fcb33116640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.650 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:03:51.453+0000 7fcb33116640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.650 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:03:57.465+0000 7fcb33116640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.650 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:03:59.473+0000 7fcb33116640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.650 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:04:01.513+0000 7fcb33116640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.650 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:04:02.541+0000 7fcb33116640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.651 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:04:03.541+0000 7fcb33116640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.651 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:04:06.553+0000 7fcb33116640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.651 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:04:07.593+0000 7fcb33116640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.651 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:04:08.557+0000 7fcb33116640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:04:09.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2024-09-13T14:04:09.654 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:03:53.689+0000 7fb2c0cf9640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.654 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:03:55.625+0000 7fb2c0cf9640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.654 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:04:02.685+0000 7fb2c0cf9640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.654 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:04:03.657+0000 7fb2c0cf9640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.654 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:04:06.657+0000 7fb2c0cf9640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-13T14:04:09.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-09-13T14:04:09.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-09-13T14:04:09.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-09-13T14:04:09.656 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-09-13T14:04:09.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-09-13T14:04:09.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:04:09.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2024-09-13T14:04:09.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-13T14:04:09.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-09-13T14:04:09.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-09-13T14:04:09.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:04:09.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2024-09-13T14:04:09.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-13T14:04:09.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-09-13T14:04:09.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-09-13T14:04:09.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:04:09.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2024-09-13T14:04:09.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-13T14:04:09.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-09-13T14:04:09.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-09-13T14:04:09.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:04:09.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2024-09-13T14:04:09.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-13T14:04:09.690 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-09-13T14:04:09.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-09-13T14:04:09.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-09-13T14:04:09.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-09-13T14:04:09.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-09-13T14:04:09.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2024-09-13T14:04:09.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:04:09.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-13T14:04:09.691 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:04:09.692 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:04:09.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:04:09.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:04:09.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:04:09.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:04:09.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:04:09.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:04:09.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:04:09.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:04:09.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:04:09.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:04:09.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:04:09.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:04:09.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:04:09.703 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:04:09.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:04:09.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:04:09.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2024-09-13T14:04:09.728 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:04:09.728 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:04:09.728 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:04:09.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37913 2024-09-13T14:04:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:04:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:04:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-09-13T14:04:09.731 INFO:tasks.workunit.client.0.smithi112.stdout:TEST PASSED 2024-09-13T14:04:09.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-09-13T14:04:09.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-09-13T14:04:09.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-09-13T14:04:09.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-09-13T14:04:09.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-09-13T14:04:09.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2024-09-13T14:04:09.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.37913 2024-09-13T14:04:09.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2024-09-13T14:04:09.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2024-09-13T14:04:09.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2024-09-13T14:04:09.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2024-09-13T14:04:09.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-recovery-scrub 2024-09-13T14:04:09.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-recovery-scrub 2024-09-13T14:04:09.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2024-09-13T14:04:09.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:04:09.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-13T14:04:09.735 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:04:09.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:04:09.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:04:09.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:04:09.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:04:09.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:04:09.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:04:09.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:04:09.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:04:09.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:04:09.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:04:09.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:04:09.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:04:09.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:04:09.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:04:09.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:04:09.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:04:09.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:04:09.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2024-09-13T14:04:09.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:04:09.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:04:09.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:04:09.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37913 2024-09-13T14:04:09.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:04:09.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:04:09.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-recovery-scrub 2024-09-13T14:04:09.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:04:09.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:04:09.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:04:09.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.37913 2024-09-13T14:04:09.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:04:09.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T14:04:09.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:04:09.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-09-13T14:04:09.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-13T14:04:09.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-recovery-scrub 2024-09-13T14:04:09.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T14:04:09.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T14:04:09.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T14:04:09.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-recovery-scrub/a 2024-09-13T14:04:09.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-13T14:04:10.180 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T14:04:10.180 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:04:10.181 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:04:10.181 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:04:10.181 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:04:10.181 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:04:10.181 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:04:10.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-13T14:04:10.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T14:04:10.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T14:04:10.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:04:10.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:04:10.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T14:04:10.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T14:04:10.400 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:04:10.400 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:04:10.400 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:04:10.401 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:04:10.401 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:04:10.401 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:04:10.401 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37913/ceph-mon.a.asok 2024-09-13T14:04:10.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:04:10.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.37913/ceph-mon.a.asok config get fsid 2024-09-13T14:04:10.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T14:04:10.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:04:10.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:04:10.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T14:04:10.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T14:04:10.518 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:04:10.518 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:04:10.518 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:04:10.518 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:04:10.518 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:04:10.518 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:04:10.518 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37913/ceph-mon.a.asok 2024-09-13T14:04:10.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:04:10.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.37913/ceph-mon.a.asok config get mon_host 2024-09-13T14:04:10.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2024-09-13T14:04:10.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-recovery-scrub 2024-09-13T14:04:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T14:04:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T14:04:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T14:04:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-recovery-scrub/x 2024-09-13T14:04:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T14:04:11.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T14:04:11.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:04:11.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:04:11.505 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:04:11.505 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:04:11.505 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:04:11.506 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:04:11.506 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:04:11.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:04:11.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0.1 ' 2024-09-13T14:04:11.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-09-13T14:04:11.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-09-13T14:04:11.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-09-13T14:04:11.541 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2024-09-13T14:04:11.542 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2024-09-13T14:04:11.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:04:11.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:04:11.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-13T14:04:11.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:04:11.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T14:04:11.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:04:11.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-09-13T14:04:11.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-13T14:04:11.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:04:11.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:04:11.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:04:11.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-09-13T14:04:11.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-09-13T14:04:11.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:04:11.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:04:11.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-13T14:04:11.547 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:04:11.547 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:04:11.547 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:04:11.547 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:04:11.548 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:04:11.548 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:04:11.548 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:04:11.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:04:11.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:04:11.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:04:11.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:04:11.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-13T14:04:11.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-13T14:04:11.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:04:11.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:04:11.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:04:11.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:04:11.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:04:11.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-09-13T14:04:11.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-09-13T14:04:11.549 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:04:11.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c2736cdb-4488-4dec-8ec6-8aa15cda0b17 2024-09-13T14:04:11.549 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 c2736cdb-4488-4dec-8ec6-8aa15cda0b17 2024-09-13T14:04:11.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c2736cdb-4488-4dec-8ec6-8aa15cda0b17' 2024-09-13T14:04:11.550 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:04:11.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBbRuRmhjjIIRAAlXyfWH8EMYN0FAkLbaR3gQ== 2024-09-13T14:04:11.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBbRuRmhjjIIRAAlXyfWH8EMYN0FAkLbaR3gQ=="}' 2024-09-13T14:04:11.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c2736cdb-4488-4dec-8ec6-8aa15cda0b17 -i td/osd-recovery-scrub/0/new.json 2024-09-13T14:04:11.799 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:04:11.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-09-13T14:04:11.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBbRuRmhjjIIRAAlXyfWH8EMYN0FAkLbaR3gQ== --osd-uuid c2736cdb-4488-4dec-8ec6-8aa15cda0b17 2024-09-13T14:04:11.855 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:11.853+0000 7fb4162577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:11.857 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:11.853+0000 7fb4162577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:11.858 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:11.857+0000 7fb4162577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:11.859 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:11.857+0000 7fb4162577c0 -1 bdev(0x555570833c00 td/osd-recovery-scrub/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:04:11.859 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:11.857+0000 7fb4162577c0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-09-13T14:04:17.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-09-13T14:04:17.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:04:17.852 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:04:17.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T14:04:17.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:04:18.268 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:04:18.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T14:04:18.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:04:18.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:04:18.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:04:18.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:04:18.305 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:18.301+0000 7fcd481487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:18.339 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:18.337+0000 7fcd481487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:18.341 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:18.337+0000 7fcd481487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:18.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T14:04:18.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:04:18.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:04:18.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:04:18.607 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:04:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:04:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:04:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:04:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:18.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:04:19.652 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:19.649+0000 7fcd481487c0 -1 Falling back to public interface 2024-09-13T14:04:19.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:04:19.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:19.943 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:04:19.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:04:19.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:19.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:04:20.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:04:21.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:04:21.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:21.263 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:04:21.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:04:21.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:21.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:04:21.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:04:22.486 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:22.485+0000 7fcd481487c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:04:22.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:04:22.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:22.580 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:04:22.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:04:22.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:22.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:04:22.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:04:23.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:04:23.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:23.987 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:04:23.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:04:23.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:23.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:04:24.323 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1612424736,v1:127.0.0.1:6803/1612424736] [v2:127.0.0.1:6804/1612424736,v1:127.0.0.1:6805/1612424736] exists,up c2736cdb-4488-4dec-8ec6-8aa15cda0b17 2024-09-13T14:04:24.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:04:24.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:04:24.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:04:24.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:04:24.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:04:24.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-13T14:04:24.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:04:24.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T14:04:24.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:04:24.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-09-13T14:04:24.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-13T14:04:24.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:04:24.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:04:24.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:04:24.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-09-13T14:04:24.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-09-13T14:04:24.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:04:24.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:04:24.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-13T14:04:24.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:04:24.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:04:24.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:04:24.328 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:04:24.328 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:04:24.328 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:04:24.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:04:24.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:04:24.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:04:24.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:04:24.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:04:24.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-13T14:04:24.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-13T14:04:24.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:04:24.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:04:24.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:04:24.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:04:24.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:04:24.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-09-13T14:04:24.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-09-13T14:04:24.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:04:24.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d76b9118-9d06-47f9-ac63-38fc6b3497f6 2024-09-13T14:04:24.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d76b9118-9d06-47f9-ac63-38fc6b3497f6' 2024-09-13T14:04:24.330 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 d76b9118-9d06-47f9-ac63-38fc6b3497f6 2024-09-13T14:04:24.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:04:24.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBoRuRmok+9FBAAmuLWu+VTDp5zyF8SOjBK9Q== 2024-09-13T14:04:24.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBoRuRmok+9FBAAmuLWu+VTDp5zyF8SOjBK9Q=="}' 2024-09-13T14:04:24.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d76b9118-9d06-47f9-ac63-38fc6b3497f6 -i td/osd-recovery-scrub/1/new.json 2024-09-13T14:04:24.735 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:04:24.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-09-13T14:04:24.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBoRuRmok+9FBAAmuLWu+VTDp5zyF8SOjBK9Q== --osd-uuid d76b9118-9d06-47f9-ac63-38fc6b3497f6 2024-09-13T14:04:24.794 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:24.793+0000 7fe098ad77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:24.820 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:24.817+0000 7fe098ad77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:24.821 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:24.817+0000 7fe098ad77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:24.821 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:24.821+0000 7fe098ad77c0 -1 bdev(0x5568c6cd5c00 td/osd-recovery-scrub/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:04:24.821 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:24.821+0000 7fe098ad77c0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-09-13T14:04:30.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-09-13T14:04:30.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:04:30.490 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:04:30.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T14:04:30.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:04:30.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T14:04:30.908 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:04:30.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:04:30.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:04:30.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:04:30.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:04:30.946 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:30.945+0000 7f303585e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:30.965 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:30.965+0000 7f303585e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:30.967 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:30.965+0000 7f303585e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:31.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T14:04:31.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:04:31.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:04:31.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:04:31.229 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:04:31.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:04:31.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:31.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:04:31.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:31.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:04:31.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:04:32.024 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:32.021+0000 7f303585e7c0 -1 Falling back to public interface 2024-09-13T14:04:32.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:04:32.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:32.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:04:32.557 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:04:32.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:32.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:04:32.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:04:33.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:04:33.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:33.887 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:04:33.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:04:33.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:33.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:04:34.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:04:34.940 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:34.937+0000 7f303585e7c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:04:35.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:04:35.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:35.223 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:04:35.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:04:35.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:35.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:04:35.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:04:36.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:04:36.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:36.567 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:04:36.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:04:36.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:36.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:04:36.896 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/819003221,v1:127.0.0.1:6811/819003221] [v2:127.0.0.1:6812/819003221,v1:127.0.0.1:6813/819003221] exists,up d76b9118-9d06-47f9-ac63-38fc6b3497f6 2024-09-13T14:04:36.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:04:36.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:04:36.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:04:36.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:04:36.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:04:36.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-13T14:04:36.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:04:36.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T14:04:36.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:04:36.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-09-13T14:04:36.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-13T14:04:36.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:04:36.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:04:36.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:04:36.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-09-13T14:04:36.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-09-13T14:04:36.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:04:36.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:04:36.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-13T14:04:36.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:04:36.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:04:36.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:04:36.901 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:04:36.901 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:04:36.902 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:04:36.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:04:36.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:04:36.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:04:36.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:04:36.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:04:36.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-13T14:04:36.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-13T14:04:36.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:04:36.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:04:36.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:04:36.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:04:36.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:04:36.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-09-13T14:04:36.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-09-13T14:04:36.903 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:04:36.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=30fb02ac-b547-48fe-85b3-e7f48bed28e6 2024-09-13T14:04:36.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 30fb02ac-b547-48fe-85b3-e7f48bed28e6' 2024-09-13T14:04:36.904 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 30fb02ac-b547-48fe-85b3-e7f48bed28e6 2024-09-13T14:04:36.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:04:36.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB0RuRmFDPtNhAA0w9L15ZgbH453NHDe+dzpQ== 2024-09-13T14:04:36.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB0RuRmFDPtNhAA0w9L15ZgbH453NHDe+dzpQ=="}' 2024-09-13T14:04:36.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 30fb02ac-b547-48fe-85b3-e7f48bed28e6 -i td/osd-recovery-scrub/2/new.json 2024-09-13T14:04:37.297 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:04:37.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-09-13T14:04:37.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQB0RuRmFDPtNhAA0w9L15ZgbH453NHDe+dzpQ== --osd-uuid 30fb02ac-b547-48fe-85b3-e7f48bed28e6 2024-09-13T14:04:37.356 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:37.353+0000 7fc9b60757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:37.373 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:37.369+0000 7fc9b60757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:37.374 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:37.373+0000 7fc9b60757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:37.375 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:37.373+0000 7fc9b60757c0 -1 bdev(0x560405c91c00 td/osd-recovery-scrub/2/block) open stat got: (1) Operation not permitted 2024-09-13T14:04:37.375 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:37.373+0000 7fc9b60757c0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-09-13T14:04:43.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-09-13T14:04:43.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:04:43.937 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T14:04:43.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T14:04:43.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:04:44.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T14:04:44.333 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T14:04:44.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:04:44.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:04:44.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:04:44.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:04:44.371 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:44.368+0000 7ff9f1cc07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:44.373 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:44.372+0000 7ff9f1cc07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:44.374 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:44.372+0000 7ff9f1cc07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:44.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T14:04:44.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:04:44.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:04:44.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:04:44.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:04:44.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:44.709 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:04:44.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:04:44.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:44.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:04:45.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:04:45.704 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:45.700+0000 7ff9f1cc07c0 -1 Falling back to public interface 2024-09-13T14:04:46.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:04:46.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:46.033 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:04:46.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:04:46.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:46.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:04:46.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:04:47.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:04:47.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:47.359 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:04:47.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:04:47.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:47.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:04:47.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:04:48.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:04:48.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:48.681 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:04:48.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:04:48.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:48.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:04:48.784 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:48.780+0000 7ff9f1cc07c0 -1 osd.2 0 log_to_monitors true 2024-09-13T14:04:49.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:04:50.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:04:50.006 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:04:50.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:50.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:04:50.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:50.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:04:50.333 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2432762869,v1:127.0.0.1:6819/2432762869] [v2:127.0.0.1:6820/2432762869,v1:127.0.0.1:6821/2432762869] exists,up 30fb02ac-b547-48fe-85b3-e7f48bed28e6 2024-09-13T14:04:50.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:04:50.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:04:50.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:04:50.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:04:50.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:04:50.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-13T14:04:50.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:04:50.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-13T14:04:50.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:04:50.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-09-13T14:04:50.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-13T14:04:50.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:04:50.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:04:50.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:04:50.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-09-13T14:04:50.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-09-13T14:04:50.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:04:50.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:04:50.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-13T14:04:50.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:04:50.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:04:50.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:04:50.338 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:04:50.338 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:04:50.338 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:04:50.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:04:50.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:04:50.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:04:50.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:04:50.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:04:50.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-13T14:04:50.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-13T14:04:50.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:04:50.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:04:50.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:04:50.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:04:50.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:04:50.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-09-13T14:04:50.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-09-13T14:04:50.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:04:50.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3dccd221-88a4-47b0-b2c3-9d62be23c5f7 2024-09-13T14:04:50.340 INFO:tasks.workunit.client.0.smithi112.stdout:add osd3 3dccd221-88a4-47b0-b2c3-9d62be23c5f7 2024-09-13T14:04:50.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 3dccd221-88a4-47b0-b2c3-9d62be23c5f7' 2024-09-13T14:04:50.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:04:50.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCCRuRmeIVTFRAA/44PPJk86AUuxhOAvoiq8A== 2024-09-13T14:04:50.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCCRuRmeIVTFRAA/44PPJk86AUuxhOAvoiq8A=="}' 2024-09-13T14:04:50.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3dccd221-88a4-47b0-b2c3-9d62be23c5f7 -i td/osd-recovery-scrub/3/new.json 2024-09-13T14:04:50.754 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:04:50.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-09-13T14:04:50.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCCRuRmeIVTFRAA/44PPJk86AUuxhOAvoiq8A== --osd-uuid 3dccd221-88a4-47b0-b2c3-9d62be23c5f7 2024-09-13T14:04:50.814 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:50.812+0000 7f6ad8d2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:50.830 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:50.828+0000 7f6ad8d2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:50.832 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:50.828+0000 7f6ad8d2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:50.832 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:50.828+0000 7f6ad8d2b7c0 -1 bdev(0x55637ee0c800 td/osd-recovery-scrub/3/block) open stat got: (1) Operation not permitted 2024-09-13T14:04:50.832 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:50.828+0000 7f6ad8d2b7c0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-09-13T14:04:57.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-09-13T14:04:57.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:04:57.011 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd3 key to auth repository 2024-09-13T14:04:57.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-13T14:04:57.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:04:57.415 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.3 2024-09-13T14:04:57.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-13T14:04:57.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:04:57.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:04:57.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:04:57.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:04:57.452 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:57.448+0000 7f97bae6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:57.487 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:57.484+0000 7f97bae6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:57.488 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:57.484+0000 7f97bae6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:04:57.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-13T14:04:57.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:04:57.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:04:57.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:04:57.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:04:57.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:57.752 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:04:57.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:04:57.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:57.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:04:58.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:04:59.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:04:59.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:04:59.078 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:04:59.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:04:59.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:04:59.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:04:59.275 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:04:59.272+0000 7f97bae6d7c0 -1 Falling back to public interface 2024-09-13T14:04:59.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:00.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:00.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:00.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:05:00.403 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:05:00.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:00.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:05:00.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:01.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:01.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:01.720 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:05:01.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:05:01.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:01.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:05:01.924 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:01.920+0000 7f97bae6d7c0 -1 osd.3 0 log_to_monitors true 2024-09-13T14:05:02.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:03.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:03.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:03.040 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:05:03.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:05:03.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:03.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:05:03.406 INFO:tasks.workunit.client.0.smithi112.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1824083987,v1:127.0.0.1:6827/1824083987] [v2:127.0.0.1:6828/1824083987,v1:127.0.0.1:6829/1824083987] exists,up 3dccd221-88a4-47b0-b2c3-9d62be23c5f7 2024-09-13T14:05:03.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:05:03.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:05:03.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:05:03.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:05:03.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:05:03.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-13T14:05:03.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:05:03.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-13T14:05:03.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:05:03.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/4 2024-09-13T14:05:03.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-13T14:05:03.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:05:03.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:05:03.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:05:03.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2024-09-13T14:05:03.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2024-09-13T14:05:03.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:05:03.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:05:03.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-13T14:05:03.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:05:03.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:05:03.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:05:03.412 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:05:03.412 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:05:03.412 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:05:03.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:05:03.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:05:03.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:05:03.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:05:03.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:05:03.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-13T14:05:03.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-13T14:05:03.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:05:03.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:05:03.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:05:03.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:05:03.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:05:03.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-09-13T14:05:03.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/4 2024-09-13T14:05:03.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:05:03.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=17ac8e89-c127-4231-9712-6023ecbbbc97 2024-09-13T14:05:03.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 17ac8e89-c127-4231-9712-6023ecbbbc97' 2024-09-13T14:05:03.415 INFO:tasks.workunit.client.0.smithi112.stdout:add osd4 17ac8e89-c127-4231-9712-6023ecbbbc97 2024-09-13T14:05:03.415 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:05:03.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCPRuRm6ncDGhAACuw9JQ4tVC96myOv8xiUBw== 2024-09-13T14:05:03.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCPRuRm6ncDGhAACuw9JQ4tVC96myOv8xiUBw=="}' 2024-09-13T14:05:03.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 17ac8e89-c127-4231-9712-6023ecbbbc97 -i td/osd-recovery-scrub/4/new.json 2024-09-13T14:05:03.802 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:05:03.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/4/new.json 2024-09-13T14:05:03.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCPRuRm6ncDGhAACuw9JQ4tVC96myOv8xiUBw== --osd-uuid 17ac8e89-c127-4231-9712-6023ecbbbc97 2024-09-13T14:05:03.862 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:03.864+0000 7f09d30a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:03.886 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:03.888+0000 7f09d30a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:03.888 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:03.888+0000 7f09d30a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:03.889 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:03.888+0000 7f09d30a67c0 -1 bdev(0x55b32c553c00 td/osd-recovery-scrub/4/block) open stat got: (1) Operation not permitted 2024-09-13T14:05:03.889 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:03.888+0000 7f09d30a67c0 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2024-09-13T14:05:10.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2024-09-13T14:05:10.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:05:10.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-13T14:05:10.083 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd4 key to auth repository 2024-09-13T14:05:10.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:05:10.495 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.4 2024-09-13T14:05:10.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-13T14:05:10.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:05:10.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:05:10.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:05:10.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:05:10.533 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:10.536+0000 7efeb86757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:10.560 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:10.560+0000 7efeb86757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:10.561 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:10.560+0000 7efeb86757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:10.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-13T14:05:10.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:05:10.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-13T14:05:10.835 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:05:10.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:05:10.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:05:10.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:10.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:05:10.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:10.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:05:11.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:11.616 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:11.616+0000 7efeb86757c0 -1 Falling back to public interface 2024-09-13T14:05:12.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:12.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:12.139 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:05:12.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:05:12.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:12.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:05:12.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:13.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:13.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:13.470 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:05:13.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:05:13.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:13.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:05:13.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:14.460 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:14.460+0000 7efeb86757c0 -1 osd.4 0 log_to_monitors true 2024-09-13T14:05:14.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:14.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:14.805 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:05:14.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:05:14.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:14.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:05:15.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:16.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:16.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:16.185 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:05:16.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:05:16.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:16.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:05:16.522 INFO:tasks.workunit.client.0.smithi112.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1353041217,v1:127.0.0.1:6835/1353041217] [v2:127.0.0.1:6836/1353041217,v1:127.0.0.1:6837/1353041217] exists,up 17ac8e89-c127-4231-9712-6023ecbbbc97 2024-09-13T14:05:16.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:05:16.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:05:16.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:05:16.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:05:16.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:05:16.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-13T14:05:16.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:05:16.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-13T14:05:16.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:05:16.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/5 2024-09-13T14:05:16.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-13T14:05:16.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:05:16.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:05:16.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:05:16.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2024-09-13T14:05:16.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2024-09-13T14:05:16.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:05:16.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:05:16.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-13T14:05:16.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:05:16.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:05:16.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:05:16.531 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:05:16.531 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:05:16.532 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:05:16.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:05:16.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:05:16.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:05:16.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:05:16.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:05:16.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-13T14:05:16.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-13T14:05:16.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:05:16.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:05:16.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:05:16.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:05:16.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:05:16.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-09-13T14:05:16.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/5 2024-09-13T14:05:16.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:05:16.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6ed5c318-4766-434b-b4f8-43fb9ae44da7 2024-09-13T14:05:16.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 6ed5c318-4766-434b-b4f8-43fb9ae44da7' 2024-09-13T14:05:16.535 INFO:tasks.workunit.client.0.smithi112.stdout:add osd5 6ed5c318-4766-434b-b4f8-43fb9ae44da7 2024-09-13T14:05:16.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:05:16.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCcRuRmJ/qnIBAAfeO+q50CjoKgIqDyX0Ci3g== 2024-09-13T14:05:16.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCcRuRmJ/qnIBAAfeO+q50CjoKgIqDyX0Ci3g=="}' 2024-09-13T14:05:16.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6ed5c318-4766-434b-b4f8-43fb9ae44da7 -i td/osd-recovery-scrub/5/new.json 2024-09-13T14:05:16.935 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:05:16.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/5/new.json 2024-09-13T14:05:16.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCcRuRmJ/qnIBAAfeO+q50CjoKgIqDyX0Ci3g== --osd-uuid 6ed5c318-4766-434b-b4f8-43fb9ae44da7 2024-09-13T14:05:16.984 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:16.984+0000 7f05483a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:17.022 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:17.024+0000 7f05483a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:17.023 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:17.024+0000 7f05483a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:17.023 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:17.024+0000 7f05483a17c0 -1 bdev(0x563ae3a01c00 td/osd-recovery-scrub/5/block) open stat got: (1) Operation not permitted 2024-09-13T14:05:17.023 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:17.024+0000 7f05483a17c0 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2024-09-13T14:05:23.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2024-09-13T14:05:23.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:05:23.112 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd5 key to auth repository 2024-09-13T14:05:23.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-13T14:05:23.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:05:23.519 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.5 2024-09-13T14:05:23.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-13T14:05:23.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:05:23.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:05:23.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:05:23.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:05:23.557 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:23.560+0000 7f890a50a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:23.578 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:23.580+0000 7f890a50a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:23.579 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:23.580+0000 7f890a50a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:23.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-13T14:05:23.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:05:23.860 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:05:23.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-13T14:05:23.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:05:23.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:05:23.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:23.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:05:23.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:23.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:05:24.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:24.896 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:24.896+0000 7f890a50a7c0 -1 Falling back to public interface 2024-09-13T14:05:25.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:25.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:25.189 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:05:25.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:05:25.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:25.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:05:25.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:26.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:26.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:26.515 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:05:26.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:05:26.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:26.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:05:26.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:27.795 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:27.796+0000 7f890a50a7c0 -1 osd.5 0 log_to_monitors true 2024-09-13T14:05:27.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:27.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:27.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:05:27.854 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:05:27.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:27.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:05:28.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:29.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:29.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:29.217 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:05:29.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:05:29.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:29.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:05:29.635 INFO:tasks.workunit.client.0.smithi112.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1679878463,v1:127.0.0.1:6843/1679878463] [v2:127.0.0.1:6844/1679878463,v1:127.0.0.1:6845/1679878463] exists,up 6ed5c318-4766-434b-b4f8-43fb9ae44da7 2024-09-13T14:05:29.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:05:29.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:05:29.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:05:29.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:05:29.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:05:29.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-13T14:05:29.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:05:29.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-13T14:05:29.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:05:29.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/6 2024-09-13T14:05:29.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-13T14:05:29.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:05:29.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:05:29.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:05:29.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2024-09-13T14:05:29.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2024-09-13T14:05:29.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:05:29.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:05:29.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-13T14:05:29.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:05:29.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:05:29.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:05:29.640 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:05:29.640 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:05:29.640 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:05:29.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:05:29.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:05:29.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:05:29.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:05:29.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:05:29.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-13T14:05:29.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-13T14:05:29.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:05:29.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:05:29.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:05:29.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:05:29.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:05:29.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-09-13T14:05:29.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/6 2024-09-13T14:05:29.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:05:29.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d0cfef01-f706-401c-a7b1-66451fb86e27 2024-09-13T14:05:29.642 INFO:tasks.workunit.client.0.smithi112.stdout:add osd6 d0cfef01-f706-401c-a7b1-66451fb86e27 2024-09-13T14:05:29.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 d0cfef01-f706-401c-a7b1-66451fb86e27' 2024-09-13T14:05:29.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:05:29.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCpRuRm+qmuJxAAhMiqsy2CVfWQZn0iyHmr7A== 2024-09-13T14:05:29.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCpRuRm+qmuJxAAhMiqsy2CVfWQZn0iyHmr7A=="}' 2024-09-13T14:05:29.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d0cfef01-f706-401c-a7b1-66451fb86e27 -i td/osd-recovery-scrub/6/new.json 2024-09-13T14:05:30.038 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:05:30.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/6/new.json 2024-09-13T14:05:30.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCpRuRm+qmuJxAAhMiqsy2CVfWQZn0iyHmr7A== --osd-uuid d0cfef01-f706-401c-a7b1-66451fb86e27 2024-09-13T14:05:30.097 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:30.096+0000 7f335f3837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:30.122 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:30.124+0000 7f335f3837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:30.124 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:30.124+0000 7f335f3837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:30.124 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:30.124+0000 7f335f3837c0 -1 bdev(0x5600f2f3bc00 td/osd-recovery-scrub/6/block) open stat got: (1) Operation not permitted 2024-09-13T14:05:30.124 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:30.124+0000 7f335f3837c0 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2024-09-13T14:05:36.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2024-09-13T14:05:36.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:05:36.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-13T14:05:36.480 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd6 key to auth repository 2024-09-13T14:05:36.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:05:36.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-13T14:05:36.911 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.6 2024-09-13T14:05:36.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:05:36.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:05:36.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:05:36.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:05:36.948 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:36.948+0000 7fc37ed487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:36.982 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:36.984+0000 7fc37ed487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:36.984 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:36.984+0000 7fc37ed487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:37.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-13T14:05:37.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:05:37.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-13T14:05:37.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:05:37.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:05:37.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:37.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:05:37.279 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:05:37.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:37.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:05:37.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:38.056+0000 7fc37ed487c0 -1 Falling back to public interface 2024-09-13T14:05:38.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:38.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:38.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:05:38.618 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:05:38.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:38.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:05:38.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:39.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:39.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:39.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:05:39.946 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:05:39.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:39.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:05:40.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:40.908 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:40.907+0000 7fc37ed487c0 -1 osd.6 0 log_to_monitors true 2024-09-13T14:05:41.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:41.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:41.312 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:05:41.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:05:41.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:41.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:05:41.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:42.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:42.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:42.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:05:42.680 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:05:42.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:42.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:05:43.013 INFO:tasks.workunit.client.0.smithi112.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1504454881,v1:127.0.0.1:6851/1504454881] [v2:127.0.0.1:6852/1504454881,v1:127.0.0.1:6853/1504454881] exists,up d0cfef01-f706-401c-a7b1-66451fb86e27 2024-09-13T14:05:43.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:05:43.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:05:43.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:05:43.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:05:43.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:05:43.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-13T14:05:43.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:05:43.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-13T14:05:43.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:05:43.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/7 2024-09-13T14:05:43.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-13T14:05:43.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:05:43.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:05:43.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:05:43.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2024-09-13T14:05:43.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2024-09-13T14:05:43.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:05:43.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:05:43.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-13T14:05:43.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:05:43.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:05:43.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:05:43.018 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:05:43.018 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:05:43.018 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:05:43.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:05:43.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' 2024-09-13T14:05:43.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:05:43.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:05:43.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:05:43.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-13T14:05:43.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-13T14:05:43.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:05:43.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:05:43.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:05:43.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:05:43.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:05:43.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-09-13T14:05:43.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/7 2024-09-13T14:05:43.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:05:43.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b6522a5d-85bf-4e0e-ba65-f41ae1092689 2024-09-13T14:05:43.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 b6522a5d-85bf-4e0e-ba65-f41ae1092689' 2024-09-13T14:05:43.020 INFO:tasks.workunit.client.0.smithi112.stdout:add osd7 b6522a5d-85bf-4e0e-ba65-f41ae1092689 2024-09-13T14:05:43.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:05:43.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC3RuRmcJWZAhAAAa4cBbf/zhNEWtE/bPF+Ww== 2024-09-13T14:05:43.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC3RuRmcJWZAhAAAa4cBbf/zhNEWtE/bPF+Ww=="}' 2024-09-13T14:05:43.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b6522a5d-85bf-4e0e-ba65-f41ae1092689 -i td/osd-recovery-scrub/7/new.json 2024-09-13T14:05:43.411 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:05:43.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/7/new.json 2024-09-13T14:05:43.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQC3RuRmcJWZAhAAAa4cBbf/zhNEWtE/bPF+Ww== --osd-uuid b6522a5d-85bf-4e0e-ba65-f41ae1092689 2024-09-13T14:05:43.472 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:43.471+0000 7f5f23bf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:43.495 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:43.495+0000 7f5f23bf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:43.496 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:43.495+0000 7f5f23bf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:43.496 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:43.495+0000 7f5f23bf77c0 -1 bdev(0x559bff53dc00 td/osd-recovery-scrub/7/block) open stat got: (1) Operation not permitted 2024-09-13T14:05:43.496 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:43.499+0000 7f5f23bf77c0 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2024-09-13T14:05:50.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2024-09-13T14:05:50.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:05:50.148 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd7 key to auth repository 2024-09-13T14:05:50.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-13T14:05:50.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:05:50.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-13T14:05:50.551 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.7 2024-09-13T14:05:50.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=9d213cdc-d63e-44d8-8631-e6442c3c13ba --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37913/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0.1 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T14:05:50.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:05:50.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:05:50.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:05:50.582 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:50.583+0000 7f80f78157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:50.584 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:50.587+0000 7f80f78157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:50.586 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:50.587+0000 7f80f78157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:05:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-13T14:05:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:05:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-13T14:05:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:05:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:05:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:05:50.893 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:05:50.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:50.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:05:51.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:51.879 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:51.879+0000 7f80f78157c0 -1 Falling back to public interface 2024-09-13T14:05:52.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:52.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:52.217 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:05:52.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:05:52.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:52.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:05:52.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:53.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:53.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:53.551 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:05:53.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:05:53.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:53.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:05:53.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:54.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:54.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:54.869 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:05:54.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:05:54.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:54.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:05:54.877 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:05:54.879+0000 7f80f78157c0 -1 osd.7 0 log_to_monitors true 2024-09-13T14:05:55.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:05:56.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:05:56.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:05:56.192 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:05:56.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:05:56.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:05:56.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:05:56.526 INFO:tasks.workunit.client.0.smithi112.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1055011732,v1:127.0.0.1:6859/1055011732] [v2:127.0.0.1:6860/1055011732,v1:127.0.0.1:6861/1055011732] exists,up b6522a5d-85bf-4e0e-ba65-f41ae1092689 2024-09-13T14:05:56.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:05:56.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:05:56.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:05:56.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2024-09-13T14:05:56.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 32 32 2024-09-13T14:05:56.957 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'test' created 2024-09-13T14:05:56.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:05:57.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2024-09-13T14:05:57.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:05:57.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:05:57.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:05:57.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:05:57.981 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:05:57.981 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:05:57.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:05:57.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:05:57.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:05:58.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:05:58.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:05:58.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:05:58.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:05:58.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:05:58.104 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:05:58.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:05:58.664 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:05:58.664 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:05:58.664 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:05:58.664 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:05:58.664 INFO:tasks.workunit.client.0.smithi112.stderr:5 2024-09-13T14:05:58.664 INFO:tasks.workunit.client.0.smithi112.stderr:6 2024-09-13T14:05:58.664 INFO:tasks.workunit.client.0.smithi112.stderr:7' 2024-09-13T14:05:58.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:05:58.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:05:58.665 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:05:58.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-13T14:05:58.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-13T14:05:58.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-13T14:05:58.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:05:58.890 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:05:59.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672978 2024-09-13T14:05:59.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672978 2024-09-13T14:05:59.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672978' 2024-09-13T14:05:59.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:05:59.040 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:05:59.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-13T14:05:59.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-13T14:05:59.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672978 2-64424509455' 2024-09-13T14:05:59.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:05:59.254 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:05:59.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345933 2024-09-13T14:05:59.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345933 2024-09-13T14:05:59.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672978 2-64424509455 3-85899345933' 2024-09-13T14:05:59.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:05:59.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:05:59.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182410 2024-09-13T14:05:59.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182410 2024-09-13T14:05:59.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672978 2-64424509455 3-85899345933 4-107374182410' 2024-09-13T14:05:59.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:05:59.669 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:05:59.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018888 2024-09-13T14:05:59.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018888 2024-09-13T14:05:59.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672978 2-64424509455 3-85899345933 4-107374182410 5-128849018888' 2024-09-13T14:05:59.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:05:59.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:06:00.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855365 2024-09-13T14:06:00.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855365 2024-09-13T14:06:00.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672978 2-64424509455 3-85899345933 4-107374182410 5-128849018888 6-150323855365' 2024-09-13T14:06:00.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:06:00.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:06:00.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691843 2024-09-13T14:06:00.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691843 2024-09-13T14:06:00.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672978 2-64424509455 3-85899345933 4-107374182410 5-128849018888 6-150323855365 7-171798691843' 2024-09-13T14:06:00.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:06:00.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-13T14:06:00.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:06:00.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:06:00.293 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-13T14:06:00.293 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:06:00.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-13T14:06:00.294 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836501 2024-09-13T14:06:00.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-13T14:06:00.294 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:06:00.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-13T14:06:00.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:06:00.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672978 2024-09-13T14:06:00.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:06:00.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:06:00.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672978 2024-09-13T14:06:00.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:06:00.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672978 2024-09-13T14:06:00.606 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672978 2024-09-13T14:06:00.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-13T14:06:00.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:06:00.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-13T14:06:00.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:06:00.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-13T14:06:00.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:06:00.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:06:00.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-13T14:06:00.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:06:00.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-13T14:06:00.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-13T14:06:00.939 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509455 2024-09-13T14:06:00.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:06:01.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-13T14:06:01.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:06:01.284 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345933 2024-09-13T14:06:01.284 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:06:01.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:06:01.287 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:06:01.287 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345933 2024-09-13T14:06:01.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345933 2024-09-13T14:06:01.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2024-09-13T14:06:01.288 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.3 seq 85899345933 2024-09-13T14:06:01.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:06:01.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345933 -lt 85899345933 2024-09-13T14:06:01.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:06:01.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182410 2024-09-13T14:06:01.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:06:01.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:06:01.669 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182410 2024-09-13T14:06:01.669 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:06:01.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182410 2024-09-13T14:06:01.670 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.4 seq 107374182410 2024-09-13T14:06:01.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182410' 2024-09-13T14:06:01.671 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:06:02.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182410 -lt 107374182410 2024-09-13T14:06:02.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:06:02.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018888 2024-09-13T14:06:02.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:06:02.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:06:02.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018888 2024-09-13T14:06:02.016 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:06:02.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018888 2024-09-13T14:06:02.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018888' 2024-09-13T14:06:02.017 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.5 seq 128849018888 2024-09-13T14:06:02.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:06:02.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018888 -lt 128849018888 2024-09-13T14:06:02.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:06:02.375 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:06:02.375 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855365 2024-09-13T14:06:02.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:06:02.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855365 2024-09-13T14:06:02.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:06:02.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855365 2024-09-13T14:06:02.378 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.6 seq 150323855365 2024-09-13T14:06:02.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855365' 2024-09-13T14:06:02.379 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:06:02.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855365 -lt 150323855365 2024-09-13T14:06:02.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:06:02.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691843 2024-09-13T14:06:02.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:06:02.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:06:02.703 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691843 2024-09-13T14:06:02.703 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:06:02.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691843 2024-09-13T14:06:02.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691843' 2024-09-13T14:06:02.705 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.7 seq 171798691843 2024-09-13T14:06:02.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:06:03.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691844 -lt 171798691843 2024-09-13T14:06:03.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:06:03.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:06:03.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:06:03.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-13T14:06:03.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:06:03.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:06:03.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:06:03.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:06:03.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:06:03.432 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:06:03.432 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:06:03.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=32 2024-09-13T14:06:03.772 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:06:03.772 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:06:03.772 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:06:04.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 = 32 2024-09-13T14:06:04.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:06:04.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:06:04.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2024-09-13T14:06:04.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2024-09-13T14:06:04.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2024-09-13T14:06:04.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2024-09-13T14:06:04.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.37913 bs=1M count=50 2024-09-13T14:06:04.667 INFO:tasks.workunit.client.0.smithi112.stderr:50+0 records in 2024-09-13T14:06:04.667 INFO:tasks.workunit.client.0.smithi112.stderr:50+0 records out 2024-09-13T14:06:04.667 INFO:tasks.workunit.client.0.smithi112.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.169035 s, 310 MB/s 2024-09-13T14:06:04.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2024-09-13T14:06:04.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:04.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.37913 2024-09-13T14:06:06.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:06.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.37913 2024-09-13T14:06:07.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:07.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.37913 2024-09-13T14:06:09.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:09.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.37913 2024-09-13T14:06:10.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:10.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.37913 2024-09-13T14:06:12.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:12.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.37913 2024-09-13T14:06:13.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:13.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.37913 2024-09-13T14:06:15.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:15.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.37913 2024-09-13T14:06:16.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:16.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.37913 2024-09-13T14:06:18.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:18.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.37913 2024-09-13T14:06:19.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:19.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.37913 2024-09-13T14:06:21.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:21.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.37913 2024-09-13T14:06:23.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:23.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.37913 2024-09-13T14:06:24.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:24.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.37913 2024-09-13T14:06:26.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:26.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.37913 2024-09-13T14:06:27.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:27.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.37913 2024-09-13T14:06:29.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:29.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.37913 2024-09-13T14:06:30.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:30.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.37913 2024-09-13T14:06:32.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:32.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.37913 2024-09-13T14:06:33.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:33.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.37913 2024-09-13T14:06:35.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:35.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.37913 2024-09-13T14:06:37.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:37.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.37913 2024-09-13T14:06:38.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:38.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.37913 2024-09-13T14:06:40.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:40.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.37913 2024-09-13T14:06:41.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:41.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.37913 2024-09-13T14:06:43.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:43.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.37913 2024-09-13T14:06:44.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:44.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.37913 2024-09-13T14:06:46.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:46.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.37913 2024-09-13T14:06:47.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:47.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.37913 2024-09-13T14:06:49.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:49.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.37913 2024-09-13T14:06:51.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:51.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.37913 2024-09-13T14:06:52.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:52.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.37913 2024-09-13T14:06:54.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:54.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.37913 2024-09-13T14:06:55.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:55.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.37913 2024-09-13T14:06:57.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:57.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.37913 2024-09-13T14:06:58.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:06:58.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.37913 2024-09-13T14:07:00.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:07:00.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.37913 2024-09-13T14:07:01.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:07:01.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.37913 2024-09-13T14:07:03.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:07:03.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.37913 2024-09-13T14:07:04.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-09-13T14:07:04.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.37913 2024-09-13T14:07:06.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.37913 2024-09-13T14:07:06.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2024-09-13T14:07:06.787 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 1 size to 3 2024-09-13T14:07:06.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2024-09-13T14:07:07.127 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T14:07:07.128 INFO:tasks.workunit.client.0.smithi112.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-09-13T14:05:58.235263+0000 50'26 50:60 [5] 5 [5] 5 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:08:24.076268+0000 0 0 2024-09-13T14:07:07.128 INFO:tasks.workunit.client.0.smithi112.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.416774+0000 50'13 50:45 [3] 3 [3] 3 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:35:51.458996+0000 0 0 2024-09-13T14:07:07.128 INFO:tasks.workunit.client.0.smithi112.stdout:1.1d 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-09-13T14:05:58.416774+0000 50'26 50:47 [1] 1 [1] 1 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:31:20.886917+0000 0 0 2024-09-13T14:07:07.128 INFO:tasks.workunit.client.0.smithi112.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.235002+0000 50'13 50:36 [1] 1 [1] 1 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:58:33.411138+0000 0 0 2024-09-13T14:07:07.128 INFO:tasks.workunit.client.0.smithi112.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.236038+0000 50'13 50:36 [1] 1 [1] 1 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:16:03.665563+0000 0 0 2024-09-13T14:07:07.128 INFO:tasks.workunit.client.0.smithi112.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T14:05:58.235663+0000 0'0 50:23 [1] 1 [1] 1 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:09:33.838428+0000 0 0 2024-09-13T14:07:07.128 INFO:tasks.workunit.client.0.smithi112.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-09-13T14:05:58.235869+0000 50'26 50:51 [5] 5 [5] 5 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:00:22.119473+0000 0 0 2024-09-13T14:07:07.128 INFO:tasks.workunit.client.0.smithi112.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.235374+0000 50'13 50:38 [6] 6 [6] 6 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:38:28.259715+0000 0 0 2024-09-13T14:07:07.129 INFO:tasks.workunit.client.0.smithi112.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T14:05:58.234423+0000 0'0 50:26 [6] 6 [6] 6 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:12:36.441179+0000 0 0 2024-09-13T14:07:07.129 INFO:tasks.workunit.client.0.smithi112.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-09-13T14:05:58.187998+0000 50'39 50:65 [4] 4 [4] 4 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:10:22.665453+0000 0 0 2024-09-13T14:07:07.129 INFO:tasks.workunit.client.0.smithi112.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-09-13T14:05:58.417107+0000 50'52 50:77 [3] 3 [3] 3 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:30:39.858562+0000 0 0 2024-09-13T14:07:07.129 INFO:tasks.workunit.client.0.smithi112.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-09-13T14:05:58.416775+0000 50'52 50:79 [4] 4 [4] 4 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:37:37.645594+0000 0 0 2024-09-13T14:07:07.129 INFO:tasks.workunit.client.0.smithi112.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.029629+0000 50'13 50:40 [0] 0 [0] 0 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:13:23.079996+0000 0 0 2024-09-13T14:07:07.129 INFO:tasks.workunit.client.0.smithi112.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T14:05:58.234423+0000 0'0 50:22 [3] 3 [3] 3 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:01:08.317227+0000 0 0 2024-09-13T14:07:07.129 INFO:tasks.workunit.client.0.smithi112.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.234835+0000 50'13 50:47 [5] 5 [5] 5 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:25:36.864820+0000 0 0 2024-09-13T14:07:07.129 INFO:tasks.workunit.client.0.smithi112.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.030137+0000 50'13 50:35 [0] 0 [0] 0 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:16:38.384196+0000 0 0 2024-09-13T14:07:07.129 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.379756+0000 50'13 50:45 [7] 7 [7] 7 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:34:07.488564+0000 0 0 2024-09-13T14:07:07.129 INFO:tasks.workunit.client.0.smithi112.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.188145+0000 50'13 50:35 [7] 7 [7] 7 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:59:03.731534+0000 0 0 2024-09-13T14:07:07.130 INFO:tasks.workunit.client.0.smithi112.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.236011+0000 50'13 50:35 [1] 1 [1] 1 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:53:16.675061+0000 0 0 2024-09-13T14:07:07.130 INFO:tasks.workunit.client.0.smithi112.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.235324+0000 50'13 50:36 [6] 6 [6] 6 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:22:58.901658+0000 0 0 2024-09-13T14:07:07.130 INFO:tasks.workunit.client.0.smithi112.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.235042+0000 50'13 50:37 [5] 5 [5] 5 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:30:09.132713+0000 0 0 2024-09-13T14:07:07.130 INFO:tasks.workunit.client.0.smithi112.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.030454+0000 50'13 50:37 [0] 0 [0] 0 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:18:57.293236+0000 0 0 2024-09-13T14:07:07.130 INFO:tasks.workunit.client.0.smithi112.stdout:1.12 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-09-13T14:05:57.826054+0000 50'26 50:49 [0] 0 [0] 0 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:26:29.244089+0000 0 0 2024-09-13T14:07:07.130 INFO:tasks.workunit.client.0.smithi112.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.416819+0000 50'13 50:35 [4] 4 [4] 4 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:54:15.532879+0000 0 0 2024-09-13T14:07:07.130 INFO:tasks.workunit.client.0.smithi112.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T14:05:58.234535+0000 0'0 50:21 [2] 2 [2] 2 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:44:23.362734+0000 0 0 2024-09-13T14:07:07.130 INFO:tasks.workunit.client.0.smithi112.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-09-13T14:05:58.235268+0000 50'39 50:60 [1] 1 [1] 1 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:08:33.567723+0000 0 0 2024-09-13T14:07:07.130 INFO:tasks.workunit.client.0.smithi112.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T14:05:58.235614+0000 0'0 50:22 [6] 6 [6] 6 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:44:13.156138+0000 0 0 2024-09-13T14:07:07.130 INFO:tasks.workunit.client.0.smithi112.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.029828+0000 50'13 50:36 [0] 0 [0] 0 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:22:53.307517+0000 0 0 2024-09-13T14:07:07.130 INFO:tasks.workunit.client.0.smithi112.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T14:05:58.234437+0000 0'0 50:23 [5] 5 [5] 5 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:03:29.613938+0000 0 0 2024-09-13T14:07:07.131 INFO:tasks.workunit.client.0.smithi112.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.234877+0000 50'13 50:34 [3] 3 [3] 3 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:19:12.722176+0000 0 0 2024-09-13T14:07:07.131 INFO:tasks.workunit.client.0.smithi112.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T14:05:58.235751+0000 0'0 50:21 [1] 1 [1] 1 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:54:37.293896+0000 0 0 2024-09-13T14:07:07.131 INFO:tasks.workunit.client.0.smithi112.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-09-13T14:05:58.030647+0000 50'13 50:36 [0] 0 [0] 0 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:06:25.833955+0000 0 0 2024-09-13T14:07:07.131 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T14:07:07.131 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T14:07:07.131 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:07:07.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:272: TEST_recovery_scrub_2: ceph tell 'osd.*' config get osd_mclock_override_recovery_settings 2024-09-13T14:07:07.283 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T14:07:07.283 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_mclock_override_recovery_settings": "false" 2024-09-13T14:07:07.283 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:07:07.295 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T14:07:07.296 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_mclock_override_recovery_settings": "false" 2024-09-13T14:07:07.296 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:07:07.308 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2: { 2024-09-13T14:07:07.308 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_mclock_override_recovery_settings": "false" 2024-09-13T14:07:07.308 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:07:07.321 INFO:tasks.workunit.client.0.smithi112.stdout:osd.3: { 2024-09-13T14:07:07.321 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_mclock_override_recovery_settings": "false" 2024-09-13T14:07:07.321 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:07:07.334 INFO:tasks.workunit.client.0.smithi112.stdout:osd.4: { 2024-09-13T14:07:07.334 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_mclock_override_recovery_settings": "false" 2024-09-13T14:07:07.334 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:07:07.347 INFO:tasks.workunit.client.0.smithi112.stdout:osd.5: { 2024-09-13T14:07:07.347 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_mclock_override_recovery_settings": "false" 2024-09-13T14:07:07.347 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:07:07.360 INFO:tasks.workunit.client.0.smithi112.stdout:osd.6: { 2024-09-13T14:07:07.360 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_mclock_override_recovery_settings": "false" 2024-09-13T14:07:07.360 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:07:07.372 INFO:tasks.workunit.client.0.smithi112.stdout:osd.7: { 2024-09-13T14:07:07.373 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_mclock_override_recovery_settings": "false" 2024-09-13T14:07:07.373 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:07:07.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2024-09-13T14:07:07.515 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T14:07:07.515 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_recovery_max_active": "0" 2024-09-13T14:07:07.516 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:07:07.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2024-09-13T14:07:07.763 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T14:07:07.763 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_recovery_max_active_hdd": "3" 2024-09-13T14:07:07.763 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:07:07.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2024-09-13T14:07:08.022 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T14:07:08.022 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_recovery_max_active_ssd": "10" 2024-09-13T14:07:08.022 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:07:08.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2024-09-13T14:07:08.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-09-13T14:07:08.037 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-09-13T14:07:08.037 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-09-13T14:07:08.037 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-09-13T14:07:08.037 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-09-13T14:07:08.487 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:07:08.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-09-13T14:07:08.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-09-13T14:07:10.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2024-09-13T14:07:10.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2024-09-13T14:07:10.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2024-09-13T14:07:10.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-09-13T14:07:10.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-09-13T14:07:10.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-09-13T14:07:10.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-09-13T14:07:10.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-09-13T14:07:10.853 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:07:10.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-09-13T14:07:10.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-09-13T14:07:12.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2024-09-13T14:07:12.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2024-09-13T14:07:12.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2024-09-13T14:07:12.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-09-13T14:07:12.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-09-13T14:07:12.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-09-13T14:07:12.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-09-13T14:07:12.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-09-13T14:07:13.188 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:07:13.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-09-13T14:07:13.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-09-13T14:07:15.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 2 -eq 10 2024-09-13T14:07:15.207 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 2 + 1 2024-09-13T14:07:15.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=3 2024-09-13T14:07:15.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-09-13T14:07:15.211 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-09-13T14:07:15.211 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-09-13T14:07:15.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-09-13T14:07:15.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-09-13T14:07:15.529 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:07:15.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 2 -ge 2 2024-09-13T14:07:15.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:288: TEST_recovery_scrub_2: break 2024-09-13T14:07:15.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:298: TEST_recovery_scrub_2: ceph pg dump pgs 2024-09-13T14:07:15.853 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T14:07:15.854 INFO:tasks.workunit.client.0.smithi112.stdout:1.1f 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-09-13T14:07:08.969762+0000 50'26 53:70 [5,0,2] 5 [5,0,2] 5 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.854 INFO:tasks.workunit.client.0.smithi112.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:09.337363+0000 50'13 53:56 [3,0,7] 3 [3,0,7] 3 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.854 INFO:tasks.workunit.client.0.smithi112.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-09-13T14:07:09.154206+0000 50'26 53:56 [1,3,0] 1 [1,3,0] 1 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.854 INFO:tasks.workunit.client.0.smithi112.stdout:1.b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:09.336985+0000 50'13 53:45 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.854 INFO:tasks.workunit.client.0.smithi112.stdout:1.a 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:09.186102+0000 50'13 53:45 [1,6,2] 1 [1,6,2] 1 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.854 INFO:tasks.workunit.client.0.smithi112.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T14:07:08.675005+0000 0'0 53:36 [1,4,5] 1 [1,4,5] 1 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T16:05:24.567380+0000 0 0 2024-09-13T14:07:15.854 INFO:tasks.workunit.client.0.smithi112.stdout:1.8 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-09-13T14:07:08.816745+0000 50'26 53:60 [5,4,3] 5 [5,4,3] 5 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.854 INFO:tasks.workunit.client.0.smithi112.stdout:1.7 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-09-13T14:07:08.629283+0000 50'13 53:49 [6,7,3] 6 [6,7,3] 6 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.854 INFO:tasks.workunit.client.0.smithi112.stdout:1.6 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:08.815744+0000 50'13 53:50 [6,7,3] 6 [6,7,3] 6 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.854 INFO:tasks.workunit.client.0.smithi112.stdout:1.5 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-09-13T14:07:09.381036+0000 50'39 53:75 [4,2,6] 4 [4,2,6] 4 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.854 INFO:tasks.workunit.client.0.smithi112.stdout:1.4 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-09-13T14:07:09.337463+0000 50'52 53:87 [3,0,6] 3 [3,0,6] 3 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.854 INFO:tasks.workunit.client.0.smithi112.stdout:1.3 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-09-13T14:07:09.002101+0000 50'52 53:88 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.854 INFO:tasks.workunit.client.0.smithi112.stdout:1.2 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:08.908793+0000 50'13 53:52 [0,7,1] 0 [0,7,1] 0 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.855 INFO:tasks.workunit.client.0.smithi112.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T14:07:08.968979+0000 0'0 53:35 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T15:00:43.565633+0000 0 0 2024-09-13T14:07:15.855 INFO:tasks.workunit.client.0.smithi112.stdout:1.1 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:09.153429+0000 50'13 53:57 [5,0,4] 5 [5,0,4] 5 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.855 INFO:tasks.workunit.client.0.smithi112.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:08.590083+0000 50'13 53:46 [0,3,4] 0 [0,3,4] 0 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.855 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:09.155221+0000 50'13 53:55 [7,0,2] 7 [7,0,2] 7 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.855 INFO:tasks.workunit.client.0.smithi112.stdout:1.d 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:08.866798+0000 50'13 53:44 [7,4,2] 7 [7,4,2] 7 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.855 INFO:tasks.workunit.client.0.smithi112.stdout:1.e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:08.674549+0000 50'13 53:44 [1,4,2] 1 [1,4,2] 1 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.855 INFO:tasks.workunit.client.0.smithi112.stdout:1.f 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:09.001220+0000 50'13 53:46 [6,0,3] 6 [6,0,3] 6 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.855 INFO:tasks.workunit.client.0.smithi112.stdout:1.10 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:09.186123+0000 50'13 53:45 [5,7,6] 5 [5,7,6] 5 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.855 INFO:tasks.workunit.client.0.smithi112.stdout:1.11 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:08.716295+0000 50'13 53:47 [0,1,3] 0 [0,1,3] 0 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.855 INFO:tasks.workunit.client.0.smithi112.stdout:1.12 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-09-13T14:07:08.589513+0000 50'26 53:60 [0,1,6] 0 [0,1,6] 0 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.855 INFO:tasks.workunit.client.0.smithi112.stdout:1.13 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:09.002251+0000 50'13 53:44 [4,6,3] 4 [4,6,3] 4 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.855 INFO:tasks.workunit.client.0.smithi112.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T14:07:09.155500+0000 0'0 53:32 [2,0,4] 2 [2,0,4] 2 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:21:01.182996+0000 0 0 2024-09-13T14:07:15.856 INFO:tasks.workunit.client.0.smithi112.stdout:1.15 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-09-13T14:07:08.816964+0000 50'39 53:69 [1,5,3] 1 [1,5,3] 1 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.856 INFO:tasks.workunit.client.0.smithi112.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T14:07:08.103491+0000 0'0 53:35 [6,0,7] 6 [6,0,7] 6 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:52:54.606317+0000 0 0 2024-09-13T14:07:15.856 INFO:tasks.workunit.client.0.smithi112.stdout:1.17 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:08.589860+0000 50'13 53:46 [0,4,5] 0 [0,4,5] 0 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.856 INFO:tasks.workunit.client.0.smithi112.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T14:07:08.629191+0000 0'0 53:34 [5,2,3] 5 [5,2,3] 5 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:38:18.044340+0000 0 0 2024-09-13T14:07:15.856 INFO:tasks.workunit.client.0.smithi112.stdout:1.19 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-09-13T14:07:09.521132+0000 50'13 53:43 [3,0,4] 3 [3,0,4] 3 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.856 INFO:tasks.workunit.client.0.smithi112.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-13T14:07:08.867004+0000 0'0 53:33 [1,7,4] 1 [1,7,4] 1 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 periodic scrub scheduled @ 2024-09-14T14:51:03.989533+0000 0 0 2024-09-13T14:07:15.856 INFO:tasks.workunit.client.0.smithi112.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-09-13T14:07:08.591204+0000 50'13 53:47 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-13T14:05:56.912116+0000 0'0 2024-09-13T14:05:56.912116+0000 0 0 no scrub is scheduled 0 0 2024-09-13T14:07:15.856 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T14:07:15.856 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T14:07:15.856 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:07:15.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pids= 2024-09-13T14:07:15.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: recov_scrub_count=0 2024-09-13T14:07:15.872 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: expr 32 - 1 2024-09-13T14:07:15.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: seq 0 31 2024-09-13T14:07:15.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:15.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 0 2024-09-13T14:07:15.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2024-09-13T14:07:15.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:15.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:15.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59812"' 2024-09-13T14:07:15.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59812' 2024-09-13T14:07:15.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:15.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.0 2024-09-13T14:07:15.878 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 1 2024-09-13T14:07:15.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59815: /' 2024-09-13T14:07:15.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2024-09-13T14:07:15.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:15.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:15.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59817"' 2024-09-13T14:07:15.878 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59817' 2024-09-13T14:07:15.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:15.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.1 2024-09-13T14:07:15.878 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 2 2024-09-13T14:07:15.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59822: /' 2024-09-13T14:07:15.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2024-09-13T14:07:15.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:15.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:15.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59824"' 2024-09-13T14:07:15.879 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59824' 2024-09-13T14:07:15.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:15.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.2 2024-09-13T14:07:15.880 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 3 2024-09-13T14:07:15.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59829: /' 2024-09-13T14:07:15.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2024-09-13T14:07:15.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:15.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:15.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59832"' 2024-09-13T14:07:15.880 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59832' 2024-09-13T14:07:15.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:15.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.3 2024-09-13T14:07:15.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59836: /' 2024-09-13T14:07:15.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 4 2024-09-13T14:07:15.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2024-09-13T14:07:15.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:15.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:15.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59840"' 2024-09-13T14:07:15.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59840' 2024-09-13T14:07:15.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:15.908 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 5 2024-09-13T14:07:15.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2024-09-13T14:07:15.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:15.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:15.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59842"' 2024-09-13T14:07:15.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59842' 2024-09-13T14:07:15.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:15.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 6 2024-09-13T14:07:15.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2024-09-13T14:07:15.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:15.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:15.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59846"' 2024-09-13T14:07:15.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59846' 2024-09-13T14:07:15.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:15.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.4 2024-09-13T14:07:15.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59845: /' 2024-09-13T14:07:15.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59850: /' 2024-09-13T14:07:15.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.5 2024-09-13T14:07:15.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.6 2024-09-13T14:07:15.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 7 2024-09-13T14:07:15.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2024-09-13T14:07:15.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:15.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:15.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59861"' 2024-09-13T14:07:15.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59861' 2024-09-13T14:07:15.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:15.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59857: /' 2024-09-13T14:07:15.942 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 8 2024-09-13T14:07:15.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2024-09-13T14:07:15.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:15.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:15.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59863"' 2024-09-13T14:07:15.943 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59863' 2024-09-13T14:07:15.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:15.952 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 9 2024-09-13T14:07:15.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2024-09-13T14:07:15.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:15.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:15.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59869"' 2024-09-13T14:07:15.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59869' 2024-09-13T14:07:15.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:15.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59866: /' 2024-09-13T14:07:15.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59868: /' 2024-09-13T14:07:15.965 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 10 2024-09-13T14:07:15.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2024-09-13T14:07:15.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:15.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:15.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59871"' 2024-09-13T14:07:15.965 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59871' 2024-09-13T14:07:15.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:15.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.8 2024-09-13T14:07:15.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.7 2024-09-13T14:07:15.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59875: /' 2024-09-13T14:07:15.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 11 2024-09-13T14:07:15.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.a 2024-09-13T14:07:15.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2024-09-13T14:07:15.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:15.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:15.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59882"' 2024-09-13T14:07:15.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59882' 2024-09-13T14:07:15.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:15.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.9 2024-09-13T14:07:15.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59878: /' 2024-09-13T14:07:15.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 12 2024-09-13T14:07:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2024-09-13T14:07:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59887"' 2024-09-13T14:07:15.992 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59887' 2024-09-13T14:07:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:15.996 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 13 2024-09-13T14:07:15.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2024-09-13T14:07:15.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:15.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:15.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59893"' 2024-09-13T14:07:15.998 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59893' 2024-09-13T14:07:15.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:15.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.b 2024-09-13T14:07:16.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59891: /' 2024-09-13T14:07:16.008 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 14 2024-09-13T14:07:16.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2024-09-13T14:07:16.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59904"' 2024-09-13T14:07:16.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59904' 2024-09-13T14:07:16.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59899: /' 2024-09-13T14:07:16.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.d 2024-09-13T14:07:16.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59901: /' 2024-09-13T14:07:16.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.c 2024-09-13T14:07:16.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 15 2024-09-13T14:07:16.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2024-09-13T14:07:16.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59908"' 2024-09-13T14:07:16.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59908' 2024-09-13T14:07:16.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 16 2024-09-13T14:07:16.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2024-09-13T14:07:16.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59912"' 2024-09-13T14:07:16.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59912' 2024-09-13T14:07:16.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.e 2024-09-13T14:07:16.030 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 17 2024-09-13T14:07:16.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2024-09-13T14:07:16.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59924"' 2024-09-13T14:07:16.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59924' 2024-09-13T14:07:16.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.10 2024-09-13T14:07:16.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 18 2024-09-13T14:07:16.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59911: /' 2024-09-13T14:07:16.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59920: /' 2024-09-13T14:07:16.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2024-09-13T14:07:16.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59929"' 2024-09-13T14:07:16.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59929' 2024-09-13T14:07:16.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59922: /' 2024-09-13T14:07:16.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.f 2024-09-13T14:07:16.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.11 2024-09-13T14:07:16.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59933: /' 2024-09-13T14:07:16.065 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 19 2024-09-13T14:07:16.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2024-09-13T14:07:16.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59940"' 2024-09-13T14:07:16.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59940' 2024-09-13T14:07:16.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 20 2024-09-13T14:07:16.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2024-09-13T14:07:16.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59942"' 2024-09-13T14:07:16.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59942' 2024-09-13T14:07:16.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.084 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 21 2024-09-13T14:07:16.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2024-09-13T14:07:16.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59949"' 2024-09-13T14:07:16.086 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59949' 2024-09-13T14:07:16.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.086 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 22 2024-09-13T14:07:16.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.14 2024-09-13T14:07:16.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2024-09-13T14:07:16.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59954: /' 2024-09-13T14:07:16.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59955"' 2024-09-13T14:07:16.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59955' 2024-09-13T14:07:16.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59952: /' 2024-09-13T14:07:16.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59939: /' 2024-09-13T14:07:16.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.12 2024-09-13T14:07:16.096 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 23 2024-09-13T14:07:16.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.13 2024-09-13T14:07:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2024-09-13T14:07:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59964"' 2024-09-13T14:07:16.098 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59964' 2024-09-13T14:07:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59958: /' 2024-09-13T14:07:16.099 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 24 2024-09-13T14:07:16.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2024-09-13T14:07:16.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59967"' 2024-09-13T14:07:16.102 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59967' 2024-09-13T14:07:16.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59970: /' 2024-09-13T14:07:16.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.18 2024-09-13T14:07:16.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.16 2024-09-13T14:07:16.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.15 2024-09-13T14:07:16.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59963: /' 2024-09-13T14:07:16.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 25 2024-09-13T14:07:16.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2024-09-13T14:07:16.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.17 2024-09-13T14:07:16.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 59991"' 2024-09-13T14:07:16.137 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 59991' 2024-09-13T14:07:16.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/59988: /' 2024-09-13T14:07:16.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 26 2024-09-13T14:07:16.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2024-09-13T14:07:16.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 60004"' 2024-09-13T14:07:16.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 60004' 2024-09-13T14:07:16.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 27 2024-09-13T14:07:16.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/60011: /' 2024-09-13T14:07:16.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/60003: /' 2024-09-13T14:07:16.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2024-09-13T14:07:16.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.19 2024-09-13T14:07:16.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 60012"' 2024-09-13T14:07:16.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 60012' 2024-09-13T14:07:16.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.192 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 28 2024-09-13T14:07:16.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/60026: /' 2024-09-13T14:07:16.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.1b 2024-09-13T14:07:16.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2024-09-13T14:07:16.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 60031"' 2024-09-13T14:07:16.194 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 60031' 2024-09-13T14:07:16.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.1a 2024-09-13T14:07:16.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 29 2024-09-13T14:07:16.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2024-09-13T14:07:16.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 60048"' 2024-09-13T14:07:16.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 60048' 2024-09-13T14:07:16.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/60046: /' 2024-09-13T14:07:16.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.1c 2024-09-13T14:07:16.240 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 30 2024-09-13T14:07:16.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.1d 2024-09-13T14:07:16.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2024-09-13T14:07:16.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 60057"' 2024-09-13T14:07:16.241 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 60057' 2024-09-13T14:07:16.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-09-13T14:07:16.253 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 31 2024-09-13T14:07:16.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2024-09-13T14:07:16.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:07:16.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:07:16.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 60065"' 2024-09-13T14:07:16.254 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 60065' 2024-09-13T14:07:16.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:306: TEST_recovery_scrub_2: wait_background_check pids 2024-09-13T14:07:16.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: pids=' 59812 59817 59824 59832 59840 59842 59846 59861 59863 59869 59871 59882 59887 59893 59904 59908 59912 59924 59929 59940 59942 59949 59955 59964 59967 59991 60004 60012 60031 60048 60057 60065' 2024-09-13T14:07:16.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: return_code=0 2024-09-13T14:07:16.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:16.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59812 2024-09-13T14:07:16.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/60053: /' 2024-09-13T14:07:16.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.1e 2024-09-13T14:07:16.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/60064: /' 2024-09-13T14:07:16.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: pg_scrub_mod 1.1f 2024-09-13T14:07:16.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/60070: /' 2024-09-13T14:07:22.662 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.7 2024-09-13T14:07:22.663 INFO:tasks.workunit.client.0.smithi112.stderr:59866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.7 2024-09-13T14:07:22.663 INFO:tasks.workunit.client.0.smithi112.stderr:59866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.7 2024-09-13T14:07:22.663 INFO:tasks.workunit.client.0.smithi112.stderr:59866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:22.663 INFO:tasks.workunit.client.0.smithi112.stderr:59866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:22.663 INFO:tasks.workunit.client.0.smithi112.stderr:59866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-09-13T14:07:22.663 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:22.663 INFO:tasks.workunit.client.0.smithi112.stderr:59866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.7 query 2024-09-13T14:07:22.663 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:22.663 INFO:tasks.workunit.client.0.smithi112.stderr:59866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:22.663 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:22.664 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:22.664 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.7 2024-09-13T14:07:22.664 INFO:tasks.workunit.client.0.smithi112.stderr:59866: instructing pg 1.7 on osd.6 to scrub 2024-09-13T14:07:22.664 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:22.664 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2024-09-13T14:07:22.664 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:22.664 INFO:tasks.workunit.client.0.smithi112.stderr:59866: dumped pgs 2024-09-13T14:07:22.664 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-09-13T14:07:22.664 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.7 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:22.664 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.7 2024-09-13T14:07:22.664 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:22.664 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:22.664 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:22.664 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:22.665 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:22.665 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:22.665 INFO:tasks.workunit.client.0.smithi112.stderr:59866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-09-13T14:07:22.665 INFO:tasks.workunit.client.0.smithi112.stderr:59866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.7 2024-09-13T14:07:22.665 INFO:tasks.workunit.client.0.smithi112.stderr:59866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:22.665 INFO:tasks.workunit.client.0.smithi112.stderr:59866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-09-13T14:07:23.865 INFO:tasks.workunit.client.0.smithi112.stderr:59866: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1 2024-09-13T14:07:23.865 INFO:tasks.workunit.client.0.smithi112.stderr:59822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1 2024-09-13T14:07:23.865 INFO:tasks.workunit.client.0.smithi112.stderr:59822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T14:07:23.865 INFO:tasks.workunit.client.0.smithi112.stderr:59822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:59822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:59822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T14:07:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:59822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:59822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1 query 2024-09-13T14:07:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1 2024-09-13T14:07:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:59822: instructing pg 1.1 on osd.5 to scrub 2024-09-13T14:07:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2024-09-13T14:07:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:59822: dumped pgs 2024-09-13T14:07:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1 2024-09-13T14:07:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:59822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-13T14:07:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:59822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T14:07:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:59822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:59822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T14:07:23.868 INFO:tasks.workunit.client.0.smithi112.stderr:59822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:23.868 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:23.868 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:23.868 INFO:tasks.workunit.client.0.smithi112.stderr:59822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:23.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:24.484 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.8 2024-09-13T14:07:24.484 INFO:tasks.workunit.client.0.smithi112.stderr:59868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.8 2024-09-13T14:07:24.484 INFO:tasks.workunit.client.0.smithi112.stderr:59868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.8 2024-09-13T14:07:24.484 INFO:tasks.workunit.client.0.smithi112.stderr:59868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:24.485 INFO:tasks.workunit.client.0.smithi112.stderr:59868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:24.485 INFO:tasks.workunit.client.0.smithi112.stderr:59868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-09-13T14:07:24.485 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:24.485 INFO:tasks.workunit.client.0.smithi112.stderr:59868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.8 query 2024-09-13T14:07:24.485 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:24.485 INFO:tasks.workunit.client.0.smithi112.stderr:59868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:24.485 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:24.485 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:24.485 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.8 2024-09-13T14:07:24.485 INFO:tasks.workunit.client.0.smithi112.stderr:59868: instructing pg 1.8 on osd.5 to scrub 2024-09-13T14:07:24.485 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:24.485 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2024-09-13T14:07:24.485 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:24.486 INFO:tasks.workunit.client.0.smithi112.stderr:59868: dumped pgs 2024-09-13T14:07:24.486 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.8 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:24.486 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.8 2024-09-13T14:07:24.486 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:24.486 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:24.486 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:24.486 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:24.486 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:24.486 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:24.486 INFO:tasks.workunit.client.0.smithi112.stderr:59868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2024-09-13T14:07:24.486 INFO:tasks.workunit.client.0.smithi112.stderr:59868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.8 2024-09-13T14:07:24.486 INFO:tasks.workunit.client.0.smithi112.stderr:59868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:24.486 INFO:tasks.workunit.client.0.smithi112.stderr:59868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-09-13T14:07:24.487 INFO:tasks.workunit.client.0.smithi112.stderr:59868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:24.487 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:24.487 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:24.487 INFO:tasks.workunit.client.0.smithi112.stderr:59868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:24.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:24.487 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.9 2024-09-13T14:07:24.487 INFO:tasks.workunit.client.0.smithi112.stderr:59875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.9 2024-09-13T14:07:24.488 INFO:tasks.workunit.client.0.smithi112.stderr:59875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.9 2024-09-13T14:07:24.488 INFO:tasks.workunit.client.0.smithi112.stderr:59875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:24.488 INFO:tasks.workunit.client.0.smithi112.stderr:59875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:24.488 INFO:tasks.workunit.client.0.smithi112.stderr:59875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-09-13T14:07:24.488 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:24.488 INFO:tasks.workunit.client.0.smithi112.stderr:59875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:24.488 INFO:tasks.workunit.client.0.smithi112.stderr:59875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.9 query 2024-09-13T14:07:24.488 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:24.488 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:24.488 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:24.488 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.9 2024-09-13T14:07:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:59875: instructing pg 1.9 on osd.1 to scrub 2024-09-13T14:07:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2024-09-13T14:07:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:59875: dumped pgs 2024-09-13T14:07:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.9 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.9 2024-09-13T14:07:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:59875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2024-09-13T14:07:24.490 INFO:tasks.workunit.client.0.smithi112.stderr:59875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.9 2024-09-13T14:07:24.490 INFO:tasks.workunit.client.0.smithi112.stderr:59875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:24.490 INFO:tasks.workunit.client.0.smithi112.stderr:59875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-09-13T14:07:24.490 INFO:tasks.workunit.client.0.smithi112.stderr:59875: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:24.490 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:07:21.241784+0000 '>' '' 2024-09-13T14:07:24.490 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:24.490 INFO:tasks.workunit.client.0.smithi112.stderr:59875: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:24.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:24.718 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1b 2024-09-13T14:07:24.718 INFO:tasks.workunit.client.0.smithi112.stderr:60026: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1b 2024-09-13T14:07:24.718 INFO:tasks.workunit.client.0.smithi112.stderr:60026: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1b 2024-09-13T14:07:24.719 INFO:tasks.workunit.client.0.smithi112.stderr:60026: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:24.719 INFO:tasks.workunit.client.0.smithi112.stderr:60026: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:24.719 INFO:tasks.workunit.client.0.smithi112.stderr:60026: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-09-13T14:07:24.719 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:24.719 INFO:tasks.workunit.client.0.smithi112.stderr:60026: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:24.719 INFO:tasks.workunit.client.0.smithi112.stderr:60026: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1b query 2024-09-13T14:07:24.719 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:24.719 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:24.719 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:24.719 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1b 2024-09-13T14:07:24.719 INFO:tasks.workunit.client.0.smithi112.stderr:60026: instructing pg 1.1b on osd.0 to scrub 2024-09-13T14:07:24.719 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:24.720 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:24.720 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2024-09-13T14:07:24.720 INFO:tasks.workunit.client.0.smithi112.stderr:60026: dumped pgs 2024-09-13T14:07:24.720 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-09-13T14:07:24.720 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1b 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:24.720 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1b 2024-09-13T14:07:24.720 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:24.720 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:24.720 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:24.720 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:24.720 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:24.720 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:24.720 INFO:tasks.workunit.client.0.smithi112.stderr:60026: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2024-09-13T14:07:24.721 INFO:tasks.workunit.client.0.smithi112.stderr:60026: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1b 2024-09-13T14:07:24.721 INFO:tasks.workunit.client.0.smithi112.stderr:60026: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:24.721 INFO:tasks.workunit.client.0.smithi112.stderr:60026: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-09-13T14:07:24.960 INFO:tasks.workunit.client.0.smithi112.stderr:60026: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: g59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.3 2024-09-13T14:07:24.961 INFO:tasks.workunit.client.0.smithi112.stderr:59836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.3 2024-09-13T14:07:24.961 INFO:tasks.workunit.client.0.smithi112.stderr:59836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T14:07:24.961 INFO:tasks.workunit.client.0.smithi112.stderr:59836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:24.961 INFO:tasks.workunit.client.0.smithi112.stderr:59836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:24.961 INFO:tasks.workunit.client.0.smithi112.stderr:59836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T14:07:24.961 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:24.961 INFO:tasks.workunit.client.0.smithi112.stderr:59836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.3 query 2024-09-13T14:07:24.961 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:24.961 INFO:tasks.workunit.client.0.smithi112.stderr:59836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:24.961 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:24.961 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:24.961 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.3 2024-09-13T14:07:24.962 INFO:tasks.workunit.client.0.smithi112.stderr:59836: instructing pg 1.3 on osd.4 to scrub 2024-09-13T14:07:24.962 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2024-09-13T14:07:24.962 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:24.962 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:24.962 INFO:tasks.workunit.client.0.smithi112.stderr:59836: dumped pgs 2024-09-13T14:07:24.962 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.3 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:24.962 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.3 2024-09-13T14:07:24.962 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:24.962 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:24.962 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:24.962 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:24.962 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:24.962 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:24.962 INFO:tasks.workunit.client.0.smithi112.stderr:59836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-13T14:07:24.963 INFO:tasks.workunit.client.0.smithi112.stderr:59836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T14:07:24.963 INFO:tasks.workunit.client.0.smithi112.stderr:59836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:24.963 INFO:tasks.workunit.client.0.smithi112.stderr:59836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:24.963 INFO:tasks.workunit.client.0.smithi112.stderr:59836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T14:07:24.963 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:24.963 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:24.963 INFO:tasks.workunit.client.0.smithi112.stderr:59836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:24.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.064 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.6 2024-09-13T14:07:25.064 INFO:tasks.workunit.client.0.smithi112.stderr:59857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.6 2024-09-13T14:07:25.065 INFO:tasks.workunit.client.0.smithi112.stderr:59857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2024-09-13T14:07:25.065 INFO:tasks.workunit.client.0.smithi112.stderr:59857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.065 INFO:tasks.workunit.client.0.smithi112.stderr:59857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.065 INFO:tasks.workunit.client.0.smithi112.stderr:59857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-09-13T14:07:25.065 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.065 INFO:tasks.workunit.client.0.smithi112.stderr:59857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:25.065 INFO:tasks.workunit.client.0.smithi112.stderr:59857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.6 query 2024-09-13T14:07:25.065 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:25.065 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:25.065 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:25.065 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.6 2024-09-13T14:07:25.065 INFO:tasks.workunit.client.0.smithi112.stderr:59857: instructing pg 1.6 on osd.6 to scrub 2024-09-13T14:07:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2024-09-13T14:07:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:59857: dumped pgs 2024-09-13T14:07:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.6 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.6 2024-09-13T14:07:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:25.067 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:25.067 INFO:tasks.workunit.client.0.smithi112.stderr:59857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-09-13T14:07:25.067 INFO:tasks.workunit.client.0.smithi112.stderr:59857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2024-09-13T14:07:25.067 INFO:tasks.workunit.client.0.smithi112.stderr:59857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.067 INFO:tasks.workunit.client.0.smithi112.stderr:59857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-09-13T14:07:25.067 INFO:tasks.workunit.client.0.smithi112.stderr:59857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.067 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:25.067 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.067 INFO:tasks.workunit.client.0.smithi112.stderr:59857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:25.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.094 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.0 2024-09-13T14:07:25.094 INFO:tasks.workunit.client.0.smithi112.stderr:59815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.0 2024-09-13T14:07:25.094 INFO:tasks.workunit.client.0.smithi112.stderr:59815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:07:25.094 INFO:tasks.workunit.client.0.smithi112.stderr:59815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.094 INFO:tasks.workunit.client.0.smithi112.stderr:59815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.094 INFO:tasks.workunit.client.0.smithi112.stderr:59815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:07:25.094 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.094 INFO:tasks.workunit.client.0.smithi112.stderr:59815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:25.094 INFO:tasks.workunit.client.0.smithi112.stderr:59815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.0 query 2024-09-13T14:07:25.095 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:25.095 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:25.095 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:25.095 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.0 2024-09-13T14:07:25.095 INFO:tasks.workunit.client.0.smithi112.stderr:59815: instructing pg 1.0 on osd.7 to scrub 2024-09-13T14:07:25.095 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2024-09-13T14:07:25.095 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:25.095 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:25.095 INFO:tasks.workunit.client.0.smithi112.stderr:59815: dumped pgs 2024-09-13T14:07:25.095 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.0 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.095 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.0 2024-09-13T14:07:25.095 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.095 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:25.095 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:25.096 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:25.096 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:25.096 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:25.096 INFO:tasks.workunit.client.0.smithi112.stderr:59815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:07:25.096 INFO:tasks.workunit.client.0.smithi112.stderr:59815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:07:25.096 INFO:tasks.workunit.client.0.smithi112.stderr:59815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.096 INFO:tasks.workunit.client.0.smithi112.stderr:59815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:07:25.096 INFO:tasks.workunit.client.0.smithi112.stderr:59815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.096 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:25.096 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.096 INFO:tasks.workunit.client.0.smithi112.stderr:59815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:25.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59817 2024-09-13T14:07:25.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59824 2024-09-13T14:07:25.176 INFO:tasks.workunit.client.0.smithi112.stderr:_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.176 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:25.176 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.176 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-09-13T14:07:25.176 INFO:tasks.workunit.client.0.smithi112.stderr:59866: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-09-13T14:07:25.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 2 2024-09-13T14:07:25.227 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.2 2024-09-13T14:07:25.227 INFO:tasks.workunit.client.0.smithi112.stderr:59829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.2 2024-09-13T14:07:25.227 INFO:tasks.workunit.client.0.smithi112.stderr:59829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.2 2024-09-13T14:07:25.227 INFO:tasks.workunit.client.0.smithi112.stderr:59829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.227 INFO:tasks.workunit.client.0.smithi112.stderr:59829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.227 INFO:tasks.workunit.client.0.smithi112.stderr:59829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-09-13T14:07:25.228 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.228 INFO:tasks.workunit.client.0.smithi112.stderr:59829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:25.228 INFO:tasks.workunit.client.0.smithi112.stderr:59829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.2 query 2024-09-13T14:07:25.228 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:25.228 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:25.228 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:25.228 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.2 2024-09-13T14:07:25.228 INFO:tasks.workunit.client.0.smithi112.stderr:59829: instructing pg 1.2 on osd.0 to scrub 2024-09-13T14:07:25.228 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:25.228 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:25.229 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2024-09-13T14:07:25.229 INFO:tasks.workunit.client.0.smithi112.stderr:59829: dumped pgs 2024-09-13T14:07:25.229 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.2 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.229 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.2 2024-09-13T14:07:25.229 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.229 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:25.229 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:25.229 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:25.229 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:25.229 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:25.230 INFO:tasks.workunit.client.0.smithi112.stderr:59829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-09-13T14:07:25.230 INFO:tasks.workunit.client.0.smithi112.stderr:59829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.2 2024-09-13T14:07:25.230 INFO:tasks.workunit.client.0.smithi112.stderr:59829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.230 INFO:tasks.workunit.client.0.smithi112.stderr:59829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-09-13T14:07:25.230 INFO:tasks.workunit.client.0.smithi112.stderr:59829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.230 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:25.230 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.230 INFO:tasks.workunit.client.0.smithi112.stderr:59829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:25.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59832 2024-09-13T14:07:25.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59840 2024-09-13T14:07:25.241 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1d 2024-09-13T14:07:25.241 INFO:tasks.workunit.client.0.smithi112.stderr:60053: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1d 2024-09-13T14:07:25.241 INFO:tasks.workunit.client.0.smithi112.stderr:60053: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1d 2024-09-13T14:07:25.241 INFO:tasks.workunit.client.0.smithi112.stderr:60053: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.242 INFO:tasks.workunit.client.0.smithi112.stderr:60053: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-09-13T14:07:25.242 INFO:tasks.workunit.client.0.smithi112.stderr:60053: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.242 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.242 INFO:tasks.workunit.client.0.smithi112.stderr:60053: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1d query 2024-09-13T14:07:25.242 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:25.242 INFO:tasks.workunit.client.0.smithi112.stderr:60053: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:25.242 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:25.242 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:25.242 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1d 2024-09-13T14:07:25.242 INFO:tasks.workunit.client.0.smithi112.stderr:60053: instructing pg 1.1d on osd.1 to scrub 2024-09-13T14:07:25.243 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:25.243 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:25.243 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2024-09-13T14:07:25.243 INFO:tasks.workunit.client.0.smithi112.stderr:60053: dumped pgs 2024-09-13T14:07:25.243 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1d 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.243 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1d 2024-09-13T14:07:25.243 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.243 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:25.243 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:25.243 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:25.243 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:25.244 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:25.244 INFO:tasks.workunit.client.0.smithi112.stderr:60053: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2024-09-13T14:07:25.244 INFO:tasks.workunit.client.0.smithi112.stderr:60053: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1d 2024-09-13T14:07:25.244 INFO:tasks.workunit.client.0.smithi112.stderr:60053: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.244 INFO:tasks.workunit.client.0.smithi112.stderr:60053: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.244 INFO:tasks.workunit.client.0.smithi112.stderr:60053: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-09-13T14:07:25.244 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:25.244 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.244 INFO:tasks.workunit.client.0.smithi112.stderr:60053: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:25.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.282 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.b 2024-09-13T14:07:25.282 INFO:tasks.workunit.client.0.smithi112.stderr:59891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.b 2024-09-13T14:07:25.282 INFO:tasks.workunit.client.0.smithi112.stderr:59891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.b 2024-09-13T14:07:25.282 INFO:tasks.workunit.client.0.smithi112.stderr:59891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:59891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-09-13T14:07:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:59891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:59891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:59891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.b query 2024-09-13T14:07:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.b 2024-09-13T14:07:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:59891: instructing pg 1.b on osd.1 to scrub 2024-09-13T14:07:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2024-09-13T14:07:25.284 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:25.284 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:25.284 INFO:tasks.workunit.client.0.smithi112.stderr:59891: dumped pgs 2024-09-13T14:07:25.284 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.b 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.284 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.b 2024-09-13T14:07:25.284 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.284 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:25.284 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:25.284 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:25.284 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:25.285 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:25.285 INFO:tasks.workunit.client.0.smithi112.stderr:59891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2024-09-13T14:07:25.285 INFO:tasks.workunit.client.0.smithi112.stderr:59891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.b 2024-09-13T14:07:25.285 INFO:tasks.workunit.client.0.smithi112.stderr:59891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.285 INFO:tasks.workunit.client.0.smithi112.stderr:59891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.285 INFO:tasks.workunit.client.0.smithi112.stderr:59891: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-09-13T14:07:25.285 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:25.285 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.285 INFO:tasks.workunit.client.0.smithi112.stderr:59891: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:25.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.405 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.18 2024-09-13T14:07:25.405 INFO:tasks.workunit.client.0.smithi112.stderr:59970: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.18 2024-09-13T14:07:25.405 INFO:tasks.workunit.client.0.smithi112.stderr:59970: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.18 2024-09-13T14:07:25.405 INFO:tasks.workunit.client.0.smithi112.stderr:59970: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.405 INFO:tasks.workunit.client.0.smithi112.stderr:59970: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.405 INFO:tasks.workunit.client.0.smithi112.stderr:59970: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-09-13T14:07:25.405 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.405 INFO:tasks.workunit.client.0.smithi112.stderr:59970: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.18 query 2024-09-13T14:07:25.405 INFO:tasks.workunit.client.0.smithi112.stderr:59970: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:25.405 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:25.405 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:25.406 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:25.406 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.18 2024-09-13T14:07:25.406 INFO:tasks.workunit.client.0.smithi112.stderr:59970: instructing pg 1.18 on osd.5 to scrub 2024-09-13T14:07:25.406 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:25.406 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2024-09-13T14:07:25.406 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:25.406 INFO:tasks.workunit.client.0.smithi112.stderr:59970: dumped pgs 2024-09-13T14:07:25.406 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.18 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.406 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.18 2024-09-13T14:07:25.406 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.406 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:25.406 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:59970: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2024-09-13T14:07:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:59970: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.18 2024-09-13T14:07:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:59970: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:59970: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:59970: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-09-13T14:07:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:07:20.911507+0000 '>' '' 2024-09-13T14:07:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:59970: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.451 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.e 2024-09-13T14:07:25.452 INFO:tasks.workunit.client.0.smithi112.stderr:59911: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.e 2024-09-13T14:07:25.452 INFO:tasks.workunit.client.0.smithi112.stderr:59911: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.e 2024-09-13T14:07:25.452 INFO:tasks.workunit.client.0.smithi112.stderr:59911: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.452 INFO:tasks.workunit.client.0.smithi112.stderr:59911: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-09-13T14:07:25.452 INFO:tasks.workunit.client.0.smithi112.stderr:59911: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.452 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.452 INFO:tasks.workunit.client.0.smithi112.stderr:59911: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:25.452 INFO:tasks.workunit.client.0.smithi112.stderr:59911: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.e query 2024-09-13T14:07:25.452 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:25.452 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:25.452 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:25.453 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.e 2024-09-13T14:07:25.453 INFO:tasks.workunit.client.0.smithi112.stderr:59911: instructing pg 1.e on osd.1 to scrub 2024-09-13T14:07:25.453 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:25.453 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:25.453 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2024-09-13T14:07:25.453 INFO:tasks.workunit.client.0.smithi112.stderr:59911: dumped pgs 2024-09-13T14:07:25.453 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.e 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.453 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.e 2024-09-13T14:07:25.453 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.453 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:25.453 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:25.453 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:59911: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2024-09-13T14:07:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:59911: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.e 2024-09-13T14:07:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:59911: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:59911: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-09-13T14:07:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:59911: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:59911: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.506 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.4 2024-09-13T14:07:25.506 INFO:tasks.workunit.client.0.smithi112.stderr:59845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.4 2024-09-13T14:07:25.506 INFO:tasks.workunit.client.0.smithi112.stderr:59845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.4 2024-09-13T14:07:25.506 INFO:tasks.workunit.client.0.smithi112.stderr:59845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.506 INFO:tasks.workunit.client.0.smithi112.stderr:59845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-09-13T14:07:25.506 INFO:tasks.workunit.client.0.smithi112.stderr:59845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.506 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.506 INFO:tasks.workunit.client.0.smithi112.stderr:59845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:25.506 INFO:tasks.workunit.client.0.smithi112.stderr:59845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.4 query 2024-09-13T14:07:25.507 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:25.507 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:25.507 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:25.507 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.4 2024-09-13T14:07:25.507 INFO:tasks.workunit.client.0.smithi112.stderr:59845: instructing pg 1.4 on osd.3 to scrub 2024-09-13T14:07:25.507 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:25.507 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:25.507 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2024-09-13T14:07:25.507 INFO:tasks.workunit.client.0.smithi112.stderr:59845: dumped pgs 2024-09-13T14:07:25.507 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.4 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.507 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.4 2024-09-13T14:07:25.507 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.507 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:25.508 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:25.508 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:25.508 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:25.508 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:25.508 INFO:tasks.workunit.client.0.smithi112.stderr:59845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-09-13T14:07:25.508 INFO:tasks.workunit.client.0.smithi112.stderr:59845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.4 2024-09-13T14:07:25.508 INFO:tasks.workunit.client.0.smithi112.stderr:59845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.508 INFO:tasks.workunit.client.0.smithi112.stderr:59845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-09-13T14:07:25.508 INFO:tasks.workunit.client.0.smithi112.stderr:59845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.508 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:25.508 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.508 INFO:tasks.workunit.client.0.smithi112.stderr:59845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:25.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59842 2024-09-13T14:07:25.563 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.d 2024-09-13T14:07:25.563 INFO:tasks.workunit.client.0.smithi112.stderr:59901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.d 2024-09-13T14:07:25.563 INFO:tasks.workunit.client.0.smithi112.stderr:59901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.d 2024-09-13T14:07:25.563 INFO:tasks.workunit.client.0.smithi112.stderr:59901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.563 INFO:tasks.workunit.client.0.smithi112.stderr:59901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-09-13T14:07:25.563 INFO:tasks.workunit.client.0.smithi112.stderr:59901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.563 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:59901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.d query 2024-09-13T14:07:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:59901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.d 2024-09-13T14:07:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:59901: instructing pg 1.d on osd.7 to scrub 2024-09-13T14:07:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2024-09-13T14:07:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:59901: dumped pgs 2024-09-13T14:07:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.d 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.d 2024-09-13T14:07:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:59901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2024-09-13T14:07:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:59901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.d 2024-09-13T14:07:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:59901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:59901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-09-13T14:07:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:59901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:59901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:25.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.573 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.c 2024-09-13T14:07:25.574 INFO:tasks.workunit.client.0.smithi112.stderr:59899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.c 2024-09-13T14:07:25.574 INFO:tasks.workunit.client.0.smithi112.stderr:59899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.c 2024-09-13T14:07:25.574 INFO:tasks.workunit.client.0.smithi112.stderr:59899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.574 INFO:tasks.workunit.client.0.smithi112.stderr:59899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.574 INFO:tasks.workunit.client.0.smithi112.stderr:59899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-09-13T14:07:25.574 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.574 INFO:tasks.workunit.client.0.smithi112.stderr:59899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.c query 2024-09-13T14:07:25.574 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:25.574 INFO:tasks.workunit.client.0.smithi112.stderr:59899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:25.574 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:25.574 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:25.574 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.c 2024-09-13T14:07:25.574 INFO:tasks.workunit.client.0.smithi112.stderr:59899: instructing pg 1.c on osd.3 to scrub 2024-09-13T14:07:25.575 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:25.575 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2024-09-13T14:07:25.575 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:25.575 INFO:tasks.workunit.client.0.smithi112.stderr:59899: dumped pgs 2024-09-13T14:07:25.575 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.c 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.575 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.c 2024-09-13T14:07:25.575 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.575 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:25.575 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:25.575 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:25.575 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:25.575 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:25.575 INFO:tasks.workunit.client.0.smithi112.stderr:59899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2024-09-13T14:07:25.575 INFO:tasks.workunit.client.0.smithi112.stderr:59899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.c 2024-09-13T14:07:25.576 INFO:tasks.workunit.client.0.smithi112.stderr:59899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.576 INFO:tasks.workunit.client.0.smithi112.stderr:59899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.576 INFO:tasks.workunit.client.0.smithi112.stderr:59899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-09-13T14:07:25.576 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:07:22.008464+0000 '>' '' 2024-09-13T14:07:25.576 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.576 INFO:tasks.workunit.client.0.smithi112.stderr:59899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:25.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.5 2024-09-13T14:07:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:59850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.5 2024-09-13T14:07:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:59850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.5 2024-09-13T14:07:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:59850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:59850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:59850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-09-13T14:07:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:59850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:59850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.5 query 2024-09-13T14:07:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.5 2024-09-13T14:07:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:59850: instructing pg 1.5 on osd.4 to scrub 2024-09-13T14:07:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2024-09-13T14:07:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:59850: dumped pgs 2024-09-13T14:07:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.5 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.5 2024-09-13T14:07:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:25.668 INFO:tasks.workunit.client.0.smithi112.stderr:59850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-09-13T14:07:25.668 INFO:tasks.workunit.client.0.smithi112.stderr:59850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.5 2024-09-13T14:07:25.668 INFO:tasks.workunit.client.0.smithi112.stderr:59850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.668 INFO:tasks.workunit.client.0.smithi112.stderr:59850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.668 INFO:tasks.workunit.client.0.smithi112.stderr:59850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-09-13T14:07:25.668 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:25.668 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.668 INFO:tasks.workunit.client.0.smithi112.stderr:59850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:25.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59846 2024-09-13T14:07:25.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59861 2024-09-13T14:07:25.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-09-13T14:07:25.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-09-13T14:07:25.675 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 0 + 1 2024-09-13T14:07:25.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=1 2024-09-13T14:07:25.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59863 2024-09-13T14:07:25.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59869 2024-09-13T14:07:25.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59871 2024-09-13T14:07:25.738 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.10 2024-09-13T14:07:25.739 INFO:tasks.workunit.client.0.smithi112.stderr:59922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.10 2024-09-13T14:07:25.739 INFO:tasks.workunit.client.0.smithi112.stderr:59922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.10 2024-09-13T14:07:25.739 INFO:tasks.workunit.client.0.smithi112.stderr:59922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.739 INFO:tasks.workunit.client.0.smithi112.stderr:59922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.739 INFO:tasks.workunit.client.0.smithi112.stderr:59922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-09-13T14:07:25.739 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.739 INFO:tasks.workunit.client.0.smithi112.stderr:59922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.10 query 2024-09-13T14:07:25.739 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:25.739 INFO:tasks.workunit.client.0.smithi112.stderr:59922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:25.739 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:25.739 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:25.740 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.10 2024-09-13T14:07:25.740 INFO:tasks.workunit.client.0.smithi112.stderr:59922: instructing pg 1.10 on osd.5 to scrub 2024-09-13T14:07:25.740 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:25.740 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:25.740 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2024-09-13T14:07:25.740 INFO:tasks.workunit.client.0.smithi112.stderr:59922: dumped pgs 2024-09-13T14:07:25.740 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.10 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.740 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.10 2024-09-13T14:07:25.740 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.740 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:25.740 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:25.740 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:25.740 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:25.740 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:25.741 INFO:tasks.workunit.client.0.smithi112.stderr:59922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2024-09-13T14:07:25.741 INFO:tasks.workunit.client.0.smithi112.stderr:59922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.10 2024-09-13T14:07:25.741 INFO:tasks.workunit.client.0.smithi112.stderr:59922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.741 INFO:tasks.workunit.client.0.smithi112.stderr:59922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.741 INFO:tasks.workunit.client.0.smithi112.stderr:59922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-09-13T14:07:25.741 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:25.741 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.741 INFO:tasks.workunit.client.0.smithi112.stderr:59922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:25.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.922 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.f 2024-09-13T14:07:25.922 INFO:tasks.workunit.client.0.smithi112.stderr:59920: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.f 2024-09-13T14:07:25.922 INFO:tasks.workunit.client.0.smithi112.stderr:59920: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.f 2024-09-13T14:07:25.922 INFO:tasks.workunit.client.0.smithi112.stderr:59920: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.922 INFO:tasks.workunit.client.0.smithi112.stderr:59920: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-09-13T14:07:25.922 INFO:tasks.workunit.client.0.smithi112.stderr:59920: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.922 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.922 INFO:tasks.workunit.client.0.smithi112.stderr:59920: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.f query 2024-09-13T14:07:25.922 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:25.922 INFO:tasks.workunit.client.0.smithi112.stderr:59920: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:25.922 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:25.922 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:25.923 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.f 2024-09-13T14:07:25.923 INFO:tasks.workunit.client.0.smithi112.stderr:59920: instructing pg 1.f on osd.6 to scrub 2024-09-13T14:07:25.923 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:25.923 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:25.923 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2024-09-13T14:07:25.923 INFO:tasks.workunit.client.0.smithi112.stderr:59920: dumped pgs 2024-09-13T14:07:25.923 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.f 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.923 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.f 2024-09-13T14:07:25.923 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.923 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:25.923 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:25.923 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:25.924 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:25.924 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:25.924 INFO:tasks.workunit.client.0.smithi112.stderr:59920: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2024-09-13T14:07:25.924 INFO:tasks.workunit.client.0.smithi112.stderr:59920: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.f 2024-09-13T14:07:25.924 INFO:tasks.workunit.client.0.smithi112.stderr:59920: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.924 INFO:tasks.workunit.client.0.smithi112.stderr:59920: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.924 INFO:tasks.workunit.client.0.smithi112.stderr:59920: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-09-13T14:07:25.924 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:25.924 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.924 INFO:tasks.workunit.client.0.smithi112.stderr:59920: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:25.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.945 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.12 2024-09-13T14:07:25.945 INFO:tasks.workunit.client.0.smithi112.stderr:59939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.12 2024-09-13T14:07:25.945 INFO:tasks.workunit.client.0.smithi112.stderr:59939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.12 2024-09-13T14:07:25.945 INFO:tasks.workunit.client.0.smithi112.stderr:59939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.945 INFO:tasks.workunit.client.0.smithi112.stderr:59939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.945 INFO:tasks.workunit.client.0.smithi112.stderr:59939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-09-13T14:07:25.945 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.946 INFO:tasks.workunit.client.0.smithi112.stderr:59939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:25.946 INFO:tasks.workunit.client.0.smithi112.stderr:59939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.12 query 2024-09-13T14:07:25.946 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:25.946 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:25.946 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:25.946 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.12 2024-09-13T14:07:25.946 INFO:tasks.workunit.client.0.smithi112.stderr:59939: instructing pg 1.12 on osd.0 to scrub 2024-09-13T14:07:25.946 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2024-09-13T14:07:25.946 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:25.946 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:25.946 INFO:tasks.workunit.client.0.smithi112.stderr:59939: dumped pgs 2024-09-13T14:07:25.946 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.12 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.946 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.12 2024-09-13T14:07:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:59939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2024-09-13T14:07:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:59939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.12 2024-09-13T14:07:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:59939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:59939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-09-13T14:07:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:59939: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:59939: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:25.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.986 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.a 2024-09-13T14:07:25.986 INFO:tasks.workunit.client.0.smithi112.stderr:59878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.a 2024-09-13T14:07:25.986 INFO:tasks.workunit.client.0.smithi112.stderr:59878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.a 2024-09-13T14:07:25.986 INFO:tasks.workunit.client.0.smithi112.stderr:59878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.986 INFO:tasks.workunit.client.0.smithi112.stderr:59878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-09-13T14:07:25.986 INFO:tasks.workunit.client.0.smithi112.stderr:59878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.986 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.986 INFO:tasks.workunit.client.0.smithi112.stderr:59878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:25.986 INFO:tasks.workunit.client.0.smithi112.stderr:59878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.a query 2024-09-13T14:07:25.986 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:25.987 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:25.987 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:25.987 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.a 2024-09-13T14:07:25.987 INFO:tasks.workunit.client.0.smithi112.stderr:59878: instructing pg 1.a on osd.1 to scrub 2024-09-13T14:07:25.987 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2024-09-13T14:07:25.987 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:25.987 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:25.987 INFO:tasks.workunit.client.0.smithi112.stderr:59878: dumped pgs 2024-09-13T14:07:25.987 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.a 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.987 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.a 2024-09-13T14:07:25.987 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:25.987 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:25.987 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:25.987 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:59878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2024-09-13T14:07:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:59878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.a 2024-09-13T14:07:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:59878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:59878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:59878: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-09-13T14:07:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:59878: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59882 2024-09-13T14:07:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59887 2024-09-13T14:07:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59893 2024-09-13T14:07:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59904 2024-09-13T14:07:25.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59908 2024-09-13T14:07:25.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59912 2024-09-13T14:07:25.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:25.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:25.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:25.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:25.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59924 2024-09-13T14:07:26.031 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1c 2024-09-13T14:07:26.031 INFO:tasks.workunit.client.0.smithi112.stderr:60046: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1c 2024-09-13T14:07:26.031 INFO:tasks.workunit.client.0.smithi112.stderr:60046: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1c 2024-09-13T14:07:26.031 INFO:tasks.workunit.client.0.smithi112.stderr:60046: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:60046: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-09-13T14:07:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:60046: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:60046: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:60046: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1c query 2024-09-13T14:07:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1c 2024-09-13T14:07:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:60046: instructing pg 1.1c on osd.0 to scrub 2024-09-13T14:07:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2024-09-13T14:07:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:60046: dumped pgs 2024-09-13T14:07:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1c 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1c 2024-09-13T14:07:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:60046: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2024-09-13T14:07:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:60046: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1c 2024-09-13T14:07:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:60046: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:60046: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-09-13T14:07:26.034 INFO:tasks.workunit.client.0.smithi112.stderr:60046: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.034 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:26.034 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:26.034 INFO:tasks.workunit.client.0.smithi112.stderr:60046: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:26.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:26.042 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.15 2024-09-13T14:07:26.043 INFO:tasks.workunit.client.0.smithi112.stderr:59958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.15 2024-09-13T14:07:26.043 INFO:tasks.workunit.client.0.smithi112.stderr:59958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.15 2024-09-13T14:07:26.043 INFO:tasks.workunit.client.0.smithi112.stderr:59958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.043 INFO:tasks.workunit.client.0.smithi112.stderr:59958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-09-13T14:07:26.043 INFO:tasks.workunit.client.0.smithi112.stderr:59958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.043 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.043 INFO:tasks.workunit.client.0.smithi112.stderr:59958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.15 query 2024-09-13T14:07:26.043 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:26.043 INFO:tasks.workunit.client.0.smithi112.stderr:59958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:26.043 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:26.043 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:26.043 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.15 2024-09-13T14:07:26.044 INFO:tasks.workunit.client.0.smithi112.stderr:59958: instructing pg 1.15 on osd.1 to scrub 2024-09-13T14:07:26.044 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:26.044 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2024-09-13T14:07:26.044 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:26.044 INFO:tasks.workunit.client.0.smithi112.stderr:59958: dumped pgs 2024-09-13T14:07:26.044 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.15 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.044 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.15 2024-09-13T14:07:26.044 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.044 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:26.044 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:26.044 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:26.044 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:26.044 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:26.045 INFO:tasks.workunit.client.0.smithi112.stderr:59958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2024-09-13T14:07:26.045 INFO:tasks.workunit.client.0.smithi112.stderr:59958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.15 2024-09-13T14:07:26.045 INFO:tasks.workunit.client.0.smithi112.stderr:59958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.045 INFO:tasks.workunit.client.0.smithi112.stderr:59958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-09-13T14:07:26.045 INFO:tasks.workunit.client.0.smithi112.stderr:59958: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.045 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:26.045 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:26.046 INFO:tasks.workunit.client.0.smithi112.stderr:59958: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:26.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:26.088 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.17 2024-09-13T14:07:26.088 INFO:tasks.workunit.client.0.smithi112.stderr:59988: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.17 2024-09-13T14:07:26.089 INFO:tasks.workunit.client.0.smithi112.stderr:59988: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.17 2024-09-13T14:07:26.089 INFO:tasks.workunit.client.0.smithi112.stderr:59988: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.089 INFO:tasks.workunit.client.0.smithi112.stderr:59988: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.089 INFO:tasks.workunit.client.0.smithi112.stderr:59988: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-09-13T14:07:26.089 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.089 INFO:tasks.workunit.client.0.smithi112.stderr:59988: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.17 query 2024-09-13T14:07:26.089 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:26.089 INFO:tasks.workunit.client.0.smithi112.stderr:59988: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:26.089 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:26.089 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:26.089 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.17 2024-09-13T14:07:26.089 INFO:tasks.workunit.client.0.smithi112.stderr:59988: instructing pg 1.17 on osd.0 to scrub 2024-09-13T14:07:26.090 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:26.090 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:26.090 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2024-09-13T14:07:26.090 INFO:tasks.workunit.client.0.smithi112.stderr:59988: dumped pgs 2024-09-13T14:07:26.090 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.17 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.090 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.17 2024-09-13T14:07:26.090 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.090 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:26.090 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:26.090 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:26.090 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:26.090 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:26.090 INFO:tasks.workunit.client.0.smithi112.stderr:59988: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2024-09-13T14:07:26.090 INFO:tasks.workunit.client.0.smithi112.stderr:59988: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.17 2024-09-13T14:07:26.091 INFO:tasks.workunit.client.0.smithi112.stderr:59988: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.091 INFO:tasks.workunit.client.0.smithi112.stderr:59988: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-09-13T14:07:26.091 INFO:tasks.workunit.client.0.smithi112.stderr:59988: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.091 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:26.091 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:26.091 INFO:tasks.workunit.client.0.smithi112.stderr:59988: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:26.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:26.092 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1a 2024-09-13T14:07:26.092 INFO:tasks.workunit.client.0.smithi112.stderr:60011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1a 2024-09-13T14:07:26.092 INFO:tasks.workunit.client.0.smithi112.stderr:60011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1a 2024-09-13T14:07:26.092 INFO:tasks.workunit.client.0.smithi112.stderr:60011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.092 INFO:tasks.workunit.client.0.smithi112.stderr:60011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-09-13T14:07:26.092 INFO:tasks.workunit.client.0.smithi112.stderr:60011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.092 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.092 INFO:tasks.workunit.client.0.smithi112.stderr:60011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:26.092 INFO:tasks.workunit.client.0.smithi112.stderr:60011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1a query 2024-09-13T14:07:26.092 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:26.093 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:26.093 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:26.093 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1a 2024-09-13T14:07:26.093 INFO:tasks.workunit.client.0.smithi112.stderr:60011: instructing pg 1.1a on osd.1 to scrub 2024-09-13T14:07:26.093 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2024-09-13T14:07:26.093 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:26.093 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:26.093 INFO:tasks.workunit.client.0.smithi112.stderr:60011: dumped pgs 2024-09-13T14:07:26.093 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1a 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.093 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1a 2024-09-13T14:07:26.093 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.093 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:26.093 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:60011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2024-09-13T14:07:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:60011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1a 2024-09-13T14:07:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:60011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:60011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:60011: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-09-13T14:07:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:60011: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:26.103 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.16 2024-09-13T14:07:26.103 INFO:tasks.workunit.client.0.smithi112.stderr:59963: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.16 2024-09-13T14:07:26.103 INFO:tasks.workunit.client.0.smithi112.stderr:59963: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.16 2024-09-13T14:07:26.104 INFO:tasks.workunit.client.0.smithi112.stderr:59963: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.104 INFO:tasks.workunit.client.0.smithi112.stderr:59963: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.104 INFO:tasks.workunit.client.0.smithi112.stderr:59963: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-09-13T14:07:26.104 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.104 INFO:tasks.workunit.client.0.smithi112.stderr:59963: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.16 query 2024-09-13T14:07:26.104 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:26.104 INFO:tasks.workunit.client.0.smithi112.stderr:59963: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:26.104 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:26.104 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:26.104 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.16 2024-09-13T14:07:26.104 INFO:tasks.workunit.client.0.smithi112.stderr:59963: instructing pg 1.16 on osd.6 to scrub 2024-09-13T14:07:26.104 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2024-09-13T14:07:26.105 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:26.105 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:26.105 INFO:tasks.workunit.client.0.smithi112.stderr:59963: dumped pgs 2024-09-13T14:07:26.105 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.16 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.105 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.16 2024-09-13T14:07:26.105 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.105 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:26.105 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:26.105 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:26.105 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:26.105 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:26.105 INFO:tasks.workunit.client.0.smithi112.stderr:59963: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2024-09-13T14:07:26.105 INFO:tasks.workunit.client.0.smithi112.stderr:59963: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.16 2024-09-13T14:07:26.106 INFO:tasks.workunit.client.0.smithi112.stderr:59963: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.106 INFO:tasks.workunit.client.0.smithi112.stderr:59963: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.106 INFO:tasks.workunit.client.0.smithi112.stderr:59963: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-09-13T14:07:26.106 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:07:21.800207+0000 '>' '' 2024-09-13T14:07:26.106 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:26.106 INFO:tasks.workunit.client.0.smithi112.stderr:59963: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:26.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:26.145 INFO:tasks.workunit.client.0.smithi112.stderr:et_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.145 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:26.145 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:26.145 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-09-13T14:07:26.146 INFO:tasks.workunit.client.0.smithi112.stderr:60026: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-09-13T14:07:26.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 2 2024-09-13T14:07:26.186 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.11 2024-09-13T14:07:26.186 INFO:tasks.workunit.client.0.smithi112.stderr:59933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.11 2024-09-13T14:07:26.186 INFO:tasks.workunit.client.0.smithi112.stderr:59933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.11 2024-09-13T14:07:26.186 INFO:tasks.workunit.client.0.smithi112.stderr:59933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.186 INFO:tasks.workunit.client.0.smithi112.stderr:59933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.186 INFO:tasks.workunit.client.0.smithi112.stderr:59933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-09-13T14:07:26.186 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.186 INFO:tasks.workunit.client.0.smithi112.stderr:59933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:26.186 INFO:tasks.workunit.client.0.smithi112.stderr:59933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.11 query 2024-09-13T14:07:26.186 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:26.186 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.11 2024-09-13T14:07:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:59933: instructing pg 1.11 on osd.0 to scrub 2024-09-13T14:07:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2024-09-13T14:07:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:59933: dumped pgs 2024-09-13T14:07:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.11 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.11 2024-09-13T14:07:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:59933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2024-09-13T14:07:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:59933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.11 2024-09-13T14:07:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:59933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:59933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:59933: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-09-13T14:07:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:59933: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:26.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:26.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59929 2024-09-13T14:07:26.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:26.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:26.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:26.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:26.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59940 2024-09-13T14:07:26.211 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.13 2024-09-13T14:07:26.211 INFO:tasks.workunit.client.0.smithi112.stderr:59954: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.13 2024-09-13T14:07:26.211 INFO:tasks.workunit.client.0.smithi112.stderr:59954: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.13 2024-09-13T14:07:26.211 INFO:tasks.workunit.client.0.smithi112.stderr:59954: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.211 INFO:tasks.workunit.client.0.smithi112.stderr:59954: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-09-13T14:07:26.211 INFO:tasks.workunit.client.0.smithi112.stderr:59954: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.211 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.211 INFO:tasks.workunit.client.0.smithi112.stderr:59954: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:26.211 INFO:tasks.workunit.client.0.smithi112.stderr:59954: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.13 query 2024-09-13T14:07:26.211 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:26.211 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:26.211 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:26.212 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.13 2024-09-13T14:07:26.212 INFO:tasks.workunit.client.0.smithi112.stderr:59954: instructing pg 1.13 on osd.4 to scrub 2024-09-13T14:07:26.212 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2024-09-13T14:07:26.212 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:26.212 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:26.212 INFO:tasks.workunit.client.0.smithi112.stderr:59954: dumped pgs 2024-09-13T14:07:26.212 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.13 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.212 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.13 2024-09-13T14:07:26.212 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.212 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:26.212 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:26.212 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:26.212 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:59954: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2024-09-13T14:07:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:59954: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.13 2024-09-13T14:07:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:59954: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:59954: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-09-13T14:07:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:59954: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:59954: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:26.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:26.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59942 2024-09-13T14:07:26.217 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.19 2024-09-13T14:07:26.217 INFO:tasks.workunit.client.0.smithi112.stderr:60003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.19 2024-09-13T14:07:26.217 INFO:tasks.workunit.client.0.smithi112.stderr:60003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.19 2024-09-13T14:07:26.217 INFO:tasks.workunit.client.0.smithi112.stderr:60003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.217 INFO:tasks.workunit.client.0.smithi112.stderr:60003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-09-13T14:07:26.218 INFO:tasks.workunit.client.0.smithi112.stderr:60003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.218 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.218 INFO:tasks.workunit.client.0.smithi112.stderr:60003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:26.218 INFO:tasks.workunit.client.0.smithi112.stderr:60003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.19 query 2024-09-13T14:07:26.218 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:26.218 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:26.218 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:26.218 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.19 2024-09-13T14:07:26.218 INFO:tasks.workunit.client.0.smithi112.stderr:60003: instructing pg 1.19 on osd.3 to scrub 2024-09-13T14:07:26.218 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2024-09-13T14:07:26.218 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:26.218 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:26.219 INFO:tasks.workunit.client.0.smithi112.stderr:60003: dumped pgs 2024-09-13T14:07:26.219 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.19 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.219 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.19 2024-09-13T14:07:26.219 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.219 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:26.219 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:26.219 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:26.219 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:26.219 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:26.219 INFO:tasks.workunit.client.0.smithi112.stderr:60003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2024-09-13T14:07:26.219 INFO:tasks.workunit.client.0.smithi112.stderr:60003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.19 2024-09-13T14:07:26.219 INFO:tasks.workunit.client.0.smithi112.stderr:60003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.219 INFO:tasks.workunit.client.0.smithi112.stderr:60003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.220 INFO:tasks.workunit.client.0.smithi112.stderr:60003: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-09-13T14:07:26.220 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:26.220 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:26.220 INFO:tasks.workunit.client.0.smithi112.stderr:60003: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:26.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:26.254 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1e 2024-09-13T14:07:26.254 INFO:tasks.workunit.client.0.smithi112.stderr:60064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1e 2024-09-13T14:07:26.254 INFO:tasks.workunit.client.0.smithi112.stderr:60064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1e 2024-09-13T14:07:26.254 INFO:tasks.workunit.client.0.smithi112.stderr:60064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.254 INFO:tasks.workunit.client.0.smithi112.stderr:60064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-09-13T14:07:26.254 INFO:tasks.workunit.client.0.smithi112.stderr:60064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.254 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.254 INFO:tasks.workunit.client.0.smithi112.stderr:60064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:26.254 INFO:tasks.workunit.client.0.smithi112.stderr:60064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1e query 2024-09-13T14:07:26.255 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:26.255 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:26.255 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:26.255 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1e 2024-09-13T14:07:26.255 INFO:tasks.workunit.client.0.smithi112.stderr:60064: instructing pg 1.1e on osd.3 to scrub 2024-09-13T14:07:26.255 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:26.255 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:26.255 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2024-09-13T14:07:26.255 INFO:tasks.workunit.client.0.smithi112.stderr:60064: dumped pgs 2024-09-13T14:07:26.255 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1e 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.255 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1e 2024-09-13T14:07:26.255 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.255 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:26.255 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:26.256 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:26.256 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:26.256 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:26.256 INFO:tasks.workunit.client.0.smithi112.stderr:60064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2024-09-13T14:07:26.256 INFO:tasks.workunit.client.0.smithi112.stderr:60064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1e 2024-09-13T14:07:26.256 INFO:tasks.workunit.client.0.smithi112.stderr:60064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.256 INFO:tasks.workunit.client.0.smithi112.stderr:60064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.256 INFO:tasks.workunit.client.0.smithi112.stderr:60064: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-09-13T14:07:26.256 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:26.256 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:26.256 INFO:tasks.workunit.client.0.smithi112.stderr:60064: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:26.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:26.277 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1f 2024-09-13T14:07:26.277 INFO:tasks.workunit.client.0.smithi112.stderr:60070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1f 2024-09-13T14:07:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:60070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1f 2024-09-13T14:07:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:60070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:60070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:60070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-09-13T14:07:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:60070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1f query 2024-09-13T14:07:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:60070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1f 2024-09-13T14:07:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:60070: instructing pg 1.1f on osd.5 to scrub 2024-09-13T14:07:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2024-09-13T14:07:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:60070: dumped pgs 2024-09-13T14:07:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1f 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1f 2024-09-13T14:07:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:60070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2024-09-13T14:07:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:60070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1f 2024-09-13T14:07:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:60070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:60070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.280 INFO:tasks.workunit.client.0.smithi112.stderr:60070: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-09-13T14:07:26.280 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:05:56.912116+0000 '>' '' 2024-09-13T14:07:26.280 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:26.280 INFO:tasks.workunit.client.0.smithi112.stderr:60070: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:26.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:26.294 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.14 2024-09-13T14:07:26.295 INFO:tasks.workunit.client.0.smithi112.stderr:59952: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.14 2024-09-13T14:07:26.295 INFO:tasks.workunit.client.0.smithi112.stderr:59952: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.14 2024-09-13T14:07:26.295 INFO:tasks.workunit.client.0.smithi112.stderr:59952: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.295 INFO:tasks.workunit.client.0.smithi112.stderr:59952: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.295 INFO:tasks.workunit.client.0.smithi112.stderr:59952: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-09-13T14:07:26.295 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.295 INFO:tasks.workunit.client.0.smithi112.stderr:59952: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-09-13T14:07:26.295 INFO:tasks.workunit.client.0.smithi112.stderr:59952: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.14 query 2024-09-13T14:07:26.295 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-09-13T14:07:26.295 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-09-13T14:07:26.295 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-09-13T14:07:26.295 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.14 2024-09-13T14:07:26.295 INFO:tasks.workunit.client.0.smithi112.stderr:59952: instructing pg 1.14 on osd.2 to scrub 2024-09-13T14:07:26.296 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-09-13T14:07:26.296 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-09-13T14:07:26.296 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2024-09-13T14:07:26.296 INFO:tasks.workunit.client.0.smithi112.stderr:59952: dumped pgs 2024-09-13T14:07:26.296 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.14 2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.296 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.14 2024-09-13T14:07:26.296 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-09-13T14:05:56.912116+0000 2024-09-13T14:07:26.296 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-09-13T14:07:26.296 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-09-13T14:07:26.296 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-09-13T14:07:26.296 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-09-13T14:07:26.296 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-09-13T14:07:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:59952: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2024-09-13T14:07:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:59952: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.14 2024-09-13T14:07:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:59952: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:07:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:59952: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:07:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:59952: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-09-13T14:07:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-09-13T14:07:23.078230+0000 '>' '' 2024-09-13T14:07:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-09-13T14:07:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:59952: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-09-13T14:07:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:07:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59949 2024-09-13T14:07:26.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:26.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:26.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:26.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:26.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59955 2024-09-13T14:07:26.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:26.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:26.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:26.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:26.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59964 2024-09-13T14:07:26.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:26.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:26.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:26.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:26.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59967 2024-09-13T14:07:26.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:26.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:26.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:26.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:26.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 59991 2024-09-13T14:07:26.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:26.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:26.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:26.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:26.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 60004 2024-09-13T14:07:26.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:26.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:26.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:26.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:26.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 60012 2024-09-13T14:07:26.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-09-13T14:07:26.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-09-13T14:07:26.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 1 + 1 2024-09-13T14:07:26.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=2 2024-09-13T14:07:26.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:26.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 60031 2024-09-13T14:07:26.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:26.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:26.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:26.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:26.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 60048 2024-09-13T14:07:26.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:26.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:26.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:26.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:26.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 60057 2024-09-13T14:07:26.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:26.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:26.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:26.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-09-13T14:07:26.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 60065 2024-09-13T14:07:26.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-09-13T14:07:26.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-09-13T14:07:26.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-09-13T14:07:26.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: eval 'pids='\'''\''' 2024-09-13T14:07:26.302 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: pids= 2024-09-13T14:07:26.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:229: wait_background_check: return 0 2024-09-13T14:07:26.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:307: TEST_recovery_scrub_2: return_code=0 2024-09-13T14:07:26.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:308: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2024-09-13T14:07:26.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: ERRORS=0 2024-09-13T14:07:26.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:311: TEST_recovery_scrub_2: test 2 -eq 0 2024-09-13T14:07:26.302 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2024-09-13T14:07:26.302 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2024-09-13T14:07:26.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/osd.5.pid 2024-09-13T14:07:26.302 INFO:tasks.workunit.client.0.smithi112.stderr:td/osd-recovery-scrub/osd.2.pid 2024-09-13T14:07:26.302 INFO:tasks.workunit.client.0.smithi112.stderr:td/osd-recovery-scrub/osd.3.pid 2024-09-13T14:07:26.303 INFO:tasks.workunit.client.0.smithi112.stderr:td/osd-recovery-scrub/mgr.x.pid 2024-09-13T14:07:26.303 INFO:tasks.workunit.client.0.smithi112.stderr:td/osd-recovery-scrub/osd.7.pid 2024-09-13T14:07:26.303 INFO:tasks.workunit.client.0.smithi112.stderr:td/osd-recovery-scrub/osd.4.pid 2024-09-13T14:07:26.303 INFO:tasks.workunit.client.0.smithi112.stderr:td/osd-recovery-scrub/mon.a.pid 2024-09-13T14:07:26.303 INFO:tasks.workunit.client.0.smithi112.stderr:td/osd-recovery-scrub/osd.1.pid 2024-09-13T14:07:26.303 INFO:tasks.workunit.client.0.smithi112.stderr:td/osd-recovery-scrub/osd.0.pid 2024-09-13T14:07:26.303 INFO:tasks.workunit.client.0.smithi112.stderr:td/osd-recovery-scrub/osd.6.pid' 2024-09-13T14:07:26.303 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.2.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.6.pid 2024-09-13T14:07:26.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: pid='54729 2024-09-13T14:07:26.303 INFO:tasks.workunit.client.0.smithi112.stderr:50491 2024-09-13T14:07:26.303 INFO:tasks.workunit.client.0.smithi112.stderr:51902 2024-09-13T14:07:26.303 INFO:tasks.workunit.client.0.smithi112.stderr:46722 2024-09-13T14:07:26.303 INFO:tasks.workunit.client.0.smithi112.stderr:57551 2024-09-13T14:07:26.303 INFO:tasks.workunit.client.0.smithi112.stderr:53316 2024-09-13T14:07:26.304 INFO:tasks.workunit.client.0.smithi112.stderr:46638 2024-09-13T14:07:26.304 INFO:tasks.workunit.client.0.smithi112.stderr:49080 2024-09-13T14:07:26.304 INFO:tasks.workunit.client.0.smithi112.stderr:47669 2024-09-13T14:07:26.304 INFO:tasks.workunit.client.0.smithi112.stderr:56140' 2024-09-13T14:07:26.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: kill -0 54729 50491 51902 46722 57551 53316 46638 49080 47669 56140 2024-09-13T14:07:26.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:327: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2024-09-13T14:07:26.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:07:26.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:07:26.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:07:26.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:07:26.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:07:31.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:07:31.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: declare -a err_strings 2024-09-13T14:07:31.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:331: TEST_recovery_scrub_2: err_strings[0]='recovery in progress.*scrubs' 2024-09-13T14:07:31.611 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: expr 8 - 1 2024-09-13T14:07:31.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: seq 0 7 2024-09-13T14:07:31.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2024-09-13T14:07:31.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2024-09-13T14:07:31.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2024-09-13T14:07:31.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2024-09-13T14:07:31.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.4.log 2024-09-13T14:07:31.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.5.log 2024-09-13T14:07:31.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.6.log 2024-09-13T14:07:31.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.7.log 2024-09-13T14:07:31.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:336: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2024-09-13T14:07:31.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:338: TEST_recovery_scrub_2: found=false 2024-09-13T14:07:31.651 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: expr 8 - 1 2024-09-13T14:07:31.652 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: seq 0 7 2024-09-13T14:07:31.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2024-09-13T14:07:31.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2024-09-13T14:07:31.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2024-09-13T14:07:31.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2024-09-13T14:07:31.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.4.log 2024-09-13T14:07:31.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.5.log 2024-09-13T14:07:31.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.6.log 2024-09-13T14:07:31.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T14:07:31.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.7.log 2024-09-13T14:07:31.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:346: TEST_recovery_scrub_2: '[' false = true ']' 2024-09-13T14:07:31.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:352: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2024-09-13T14:07:31.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2024-09-13T14:07:31.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:07:31.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-13T14:07:31.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:07:31.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:07:31.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:07:31.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:07:31.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:07:31.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:07:31.706 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:07:31.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:07:31.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:07:31.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:07:31.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:07:31.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:07:31.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:07:31.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:07:31.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:07:31.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:07:31.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:07:31.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:07:31.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2024-09-13T14:07:31.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:07:31.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:07:31.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:07:31.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37913 2024-09-13T14:07:31.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:07:31.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:07:31.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:354: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2024-09-13T14:07:31.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:360: TEST_recovery_scrub_2: echo 'TEST PASSED' 2024-09-13T14:07:31.741 INFO:tasks.workunit.client.0.smithi112.stdout:TEST PASSED 2024-09-13T14:07:31.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:361: TEST_recovery_scrub_2: return 0 2024-09-13T14:07:31.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-13T14:07:31.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-recovery-scrub 0 2024-09-13T14:07:31.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2024-09-13T14:07:31.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-13T14:07:31.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-13T14:07:31.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:07:31.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:07:31.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:07:31.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:07:31.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:07:31.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:07:31.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:07:31.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:07:31.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:07:31.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:07:31.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:07:31.745 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:07:31.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:07:31.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:07:31.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:07:31.747 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:07:31.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:07:31.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-13T14:07:31.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2024-09-13T14:07:31.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:07:31.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:07:31.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37913 2024-09-13T14:07:31.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37913 2024-09-13T14:07:31.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:07:31.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:07:31.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-13T14:07:31.751 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T14:07:31.751 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T14:07:31.761 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-09-13T14:07:31.761 DEBUG:teuthology.orchestra.run.smithi112:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d58f73069a8157a9ace12a0247ee3e8497707ea9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-09-13T14:07:31.812 INFO:tasks.workunit.client.0.smithi112.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-13T14:07:31.817 INFO:tasks.workunit.client.0.smithi112.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-13T14:07:31.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-13T14:07:31.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-13T14:07:31.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-13T14:07:31.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-13T14:07:31.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-13T14:07:31.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-13T14:07:31.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-13T14:07:31.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-13T14:07:31.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-scrub-dump 2024-09-13T14:07:31.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-09-13T14:07:31.818 INFO:tasks.workunit.client.0.smithi112.stdout:This test is disabled 2024-09-13T14:07:31.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-09-13T14:07:31.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-13T14:07:31.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-scrub-dump 0 2024-09-13T14:07:31.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-dump 2024-09-13T14:07:31.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-13T14:07:31.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-dump KILL 2024-09-13T14:07:31.820 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:07:31.820 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:07:31.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:07:31.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:07:31.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:07:31.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:07:31.821 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:07:31.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:07:31.822 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:07:31.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:07:31.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:07:31.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:07:31.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:07:31.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:07:31.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:07:31.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:07:31.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:07:31.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-13T14:07:31.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-dump 2024-09-13T14:07:31.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:07:31.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:07:31.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63258 2024-09-13T14:07:31.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63258 2024-09-13T14:07:31.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:07:31.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:07:31.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-13T14:07:31.829 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T14:07:31.829 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T14:07:31.882 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-09-13T14:07:31.882 DEBUG:teuthology.orchestra.run.smithi112:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d58f73069a8157a9ace12a0247ee3e8497707ea9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-09-13T14:07:31.934 INFO:tasks.workunit.client.0.smithi112.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-09-13T14:07:31.934 INFO:tasks.workunit.client.0.smithi112.stderr:++ TIMEOUT=300 2024-09-13T14:07:31.934 INFO:tasks.workunit.client.0.smithi112.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-13T14:07:31.934 INFO:tasks.workunit.client.0.smithi112.stderr:++ MAX_TIMEOUT=15 2024-09-13T14:07:31.934 INFO:tasks.workunit.client.0.smithi112.stderr:++ PG_NUM=4 2024-09-13T14:07:31.934 INFO:tasks.workunit.client.0.smithi112.stderr:++ TMPDIR=/tmp 2024-09-13T14:07:31.934 INFO:tasks.workunit.client.0.smithi112.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-13T14:07:31.934 INFO:tasks.workunit.client.0.smithi112.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-13T14:07:31.934 INFO:tasks.workunit.client.0.smithi112.stderr:+++ uname 2024-09-13T14:07:31.935 INFO:tasks.workunit.client.0.smithi112.stderr:++ '[' Linux = FreeBSD ']' 2024-09-13T14:07:31.935 INFO:tasks.workunit.client.0.smithi112.stderr:++ SED=sed 2024-09-13T14:07:31.935 INFO:tasks.workunit.client.0.smithi112.stderr:++ AWK=awk 2024-09-13T14:07:31.936 INFO:tasks.workunit.client.0.smithi112.stderr:+++ stty -a 2024-09-13T14:07:31.936 INFO:tasks.workunit.client.0.smithi112.stderr:+++ head -1 2024-09-13T14:07:31.936 INFO:tasks.workunit.client.0.smithi112.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-13T14:07:31.937 INFO:tasks.workunit.client.0.smithi112.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-13T14:07:31.938 INFO:tasks.workunit.client.0.smithi112.stderr:++ termwidth= 2024-09-13T14:07:31.938 INFO:tasks.workunit.client.0.smithi112.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-13T14:07:31.938 INFO:tasks.workunit.client.0.smithi112.stderr:++ DIFFCOLOPTS='-y ' 2024-09-13T14:07:31.938 INFO:tasks.workunit.client.0.smithi112.stderr:++ KERNCORE=kernel.core_pattern 2024-09-13T14:07:31.938 INFO:tasks.workunit.client.0.smithi112.stderr:++ EXTRA_OPTS= 2024-09-13T14:07:31.942 INFO:tasks.workunit.client.0.smithi112.stderr:++ test '' = TESTS 2024-09-13T14:07:31.942 INFO:tasks.workunit.client.0.smithi112.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-09-13T14:07:31.943 INFO:tasks.workunit.client.0.smithi112.stderr:++ uname 2024-09-13T14:07:31.944 INFO:tasks.workunit.client.0.smithi112.stderr:+ '[' Linux = FreeBSD ']' 2024-09-13T14:07:31.945 INFO:tasks.workunit.client.0.smithi112.stderr:+ use_ec_overwrite=true 2024-09-13T14:07:31.945 INFO:tasks.workunit.client.0.smithi112.stderr:+ getjson=no 2024-09-13T14:07:31.945 INFO:tasks.workunit.client.0.smithi112.stderr:+ jqfilter='def walk(f): 2024-09-13T14:07:31.945 INFO:tasks.workunit.client.0.smithi112.stderr: . as $in 2024-09-13T14:07:31.945 INFO:tasks.workunit.client.0.smithi112.stderr: | if type == "object" then 2024-09-13T14:07:31.945 INFO:tasks.workunit.client.0.smithi112.stderr: reduce keys[] as $key 2024-09-13T14:07:31.945 INFO:tasks.workunit.client.0.smithi112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-13T14:07:31.945 INFO:tasks.workunit.client.0.smithi112.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-13T14:07:31.945 INFO:tasks.workunit.client.0.smithi112.stderr: else f 2024-09-13T14:07:31.945 INFO:tasks.workunit.client.0.smithi112.stderr: end; 2024-09-13T14:07:31.945 INFO:tasks.workunit.client.0.smithi112.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-13T14:07:31.945 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-13T14:07:31.945 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-13T14:07:31.946 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-13T14:07:31.946 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-13T14:07:31.946 INFO:tasks.workunit.client.0.smithi112.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-13T14:07:31.952 INFO:tasks.workunit.client.0.smithi112.stderr:+ main osd-scrub-repair 2024-09-13T14:07:31.952 INFO:tasks.workunit.client.0.smithi112.stderr:+ local dir=td/osd-scrub-repair 2024-09-13T14:07:31.952 INFO:tasks.workunit.client.0.smithi112.stderr:+ shift 2024-09-13T14:07:31.952 INFO:tasks.workunit.client.0.smithi112.stderr:+ shopt -s -o xtrace 2024-09-13T14:07:31.952 INFO:tasks.workunit.client.0.smithi112.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-13T14:07:31.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-13T14:07:31.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-13T14:07:31.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-13T14:07:31.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-13T14:07:31.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-13T14:07:31.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-13T14:07:31.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-13T14:07:31.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-13T14:07:31.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-scrub-repair 2024-09-13T14:07:31.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-09-13T14:07:31.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-09-13T14:07:31.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-09-13T14:07:31.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-09-13T14:07:31.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-09-13T14:07:31.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-09-13T14:07:31.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none ' 2024-09-13T14:07:31.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-09-13T14:07:31.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-09-13T14:07:31.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-13T14:07:31.954 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-09-13T14:07:31.954 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-13T14:07:31.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-09-13T14:07:31.958 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_auto_repair_bluestore_basic 2024-09-13T14:07:31.958 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_auto_repair_bluestore_failed 2024-09-13T14:07:31.958 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-09-13T14:07:31.959 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_auto_repair_bluestore_scrub 2024-09-13T14:07:31.959 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_auto_repair_bluestore_tag 2024-09-13T14:07:31.959 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_auto_repair_erasure_coded_appends 2024-09-13T14:07:31.959 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-09-13T14:07:31.959 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-09-13T14:07:31.959 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-09-13T14:07:31.959 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_corrupt_and_repair_lrc_appends 2024-09-13T14:07:31.959 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-09-13T14:07:31.959 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_corrupt_and_repair_replicated 2024-09-13T14:07:31.959 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_corrupt_scrub_erasure_appends 2024-09-13T14:07:31.959 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-09-13T14:07:31.959 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_corrupt_scrub_replicated 2024-09-13T14:07:31.959 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_corrupt_snapset_scrub_rep 2024-09-13T14:07:31.959 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_list_missing_erasure_coded_appends 2024-09-13T14:07:31.960 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_list_missing_erasure_coded_overwrites 2024-09-13T14:07:31.960 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_periodic_scrub_replicated 2024-09-13T14:07:31.960 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_repair_stats 2024-09-13T14:07:31.960 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_repair_stats_ec 2024-09-13T14:07:31.960 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_request_scrub_priority 2024-09-13T14:07:31.960 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_scrub_warning 2024-09-13T14:07:31.960 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_skip_non_repair_during_recovery 2024-09-13T14:07:31.960 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_unfound_erasure_coded_appends 2024-09-13T14:07:31.960 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_unfound_erasure_coded_overwrites' 2024-09-13T14:07:31.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T14:07:31.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T14:07:31.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T14:07:31.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T14:07:31.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:07:31.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:07:31.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:07:31.961 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:07:31.961 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:07:31.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:07:31.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:07:31.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:07:31.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:07:31.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:07:31.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:07:31.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:07:31.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:07:31.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:07:31.963 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:07:31.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:07:31.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:07:31.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:07:31.965 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:07:31.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:07:31.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:07:31.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:07:31.969 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:07:31.969 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:07:31.969 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:07:31.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:07:31.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:07:31.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:07:31.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T14:07:31.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:07:31.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:07:31.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:07:31.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T14:07:31.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:07:31.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-13T14:07:31.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-13T14:07:31.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:07:31.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T14:07:31.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-09-13T14:07:31.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-09-13T14:07:31.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-09-13T14:07:31.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-13T14:07:31.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T14:07:31.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T14:07:31.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T14:07:31.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T14:07:31.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T14:07:31.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-13T14:07:32.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T14:07:32.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:07:32.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:07:32.417 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:07:32.417 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:07:32.417 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:07:32.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:07:32.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-13T14:07:32.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T14:07:32.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T14:07:32.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:07:32.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:07:32.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T14:07:32.621 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T14:07:32.621 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:07:32.621 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:07:32.621 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:07:32.622 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:07:32.622 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:07:32.622 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:07:32.622 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:07:32.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:07:32.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T14:07:32.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T14:07:32.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:07:32.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:07:32.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T14:07:32.743 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:07:32.743 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:07:32.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T14:07:32.744 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:07:32.744 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:07:32.744 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:07:32.744 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:07:32.744 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:07:32.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:07:32.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T14:07:32.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-09-13T14:07:32.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T14:07:32.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T14:07:32.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T14:07:32.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T14:07:32.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T14:07:32.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T14:07:33.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T14:07:33.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:07:33.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:07:33.727 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:07:33.727 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:07:33.727 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:07:33.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:07:33.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:07:33.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:07:33.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-13T14:07:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:07:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:07:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T14:07:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:07:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:07:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:07:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:07:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:07:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:07:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:07:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:07:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:07:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:07:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:07:33.768 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:07:33.768 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:07:33.768 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:07:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:07:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:07:33.769 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:07:33.769 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:07:33.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:07:33.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:07:33.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:07:33.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:07:33.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:07:33.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:07:33.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:07:33.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:07:33.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:07:33.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:07:33.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:07:33.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-13T14:07:33.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:07:33.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:07:33.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=346b2060-e421-4790-83a4-b832b5bf4f69 2024-09-13T14:07:33.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 346b2060-e421-4790-83a4-b832b5bf4f69' 2024-09-13T14:07:33.771 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 346b2060-e421-4790-83a4-b832b5bf4f69 2024-09-13T14:07:33.771 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:07:33.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAlR+Rmp1EJLxAAwd/ZJ85KFU0NmCP7CpBVBg== 2024-09-13T14:07:33.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAlR+Rmp1EJLxAAwd/ZJ85KFU0NmCP7CpBVBg=="}' 2024-09-13T14:07:33.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 346b2060-e421-4790-83a4-b832b5bf4f69 -i td/osd-scrub-repair/0/new.json 2024-09-13T14:07:34.030 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:07:34.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T14:07:34.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQAlR+Rmp1EJLxAAwd/ZJ85KFU0NmCP7CpBVBg== --osd-uuid 346b2060-e421-4790-83a4-b832b5bf4f69 2024-09-13T14:07:34.087 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:07:34.086+0000 7f9439fdb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:07:34.089 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:07:34.090+0000 7f9439fdb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:07:34.090 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:07:34.090+0000 7f9439fdb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:07:34.090 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:07:34.090+0000 7f9439fdb7c0 -1 bdev(0x556c06b69c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:07:34.090 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:07:34.090+0000 7f9439fdb7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T14:07:40.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T14:07:40.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:07:40.199 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:07:40.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T14:07:40.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:07:40.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T14:07:40.597 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:07:40.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:07:40.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-13T14:07:40.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:07:40.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:07:40.632 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:07:40.634+0000 7fa7c5b3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:07:40.637 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:07:40.638+0000 7fa7c5b3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:07:40.638 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:07:40.638+0000 7fa7c5b3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:07:40.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T14:07:40.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:07:40.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:07:40.928 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:07:40.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:07:40.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:07:40.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:07:40.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:07:40.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:07:40.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:07:41.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:07:42.154 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:07:42.154+0000 7fa7c5b3b7c0 -1 Falling back to public interface 2024-09-13T14:07:42.256 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:07:42.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:07:42.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:07:42.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:07:42.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:07:42.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:07:42.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:07:43.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:07:43.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:07:43.576 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:07:43.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:07:43.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:07:43.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:07:43.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:07:44.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:07:44.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:07:44.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:07:44.900 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:07:44.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:07:44.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:07:45.020 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:07:45.022+0000 7fa7c5b3b7c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:07:45.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:07:46.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:07:46.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:07:46.220 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:07:46.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:07:46.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:07:46.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:07:46.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:07:47.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:07:47.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:07:47.598 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:07:47.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:07:47.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:07:47.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:07:47.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:07:48.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:07:48.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:07:48.981 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:07:48.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:07:48.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:07:48.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:07:49.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:07:50.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:07:50.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:07:50.309 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:07:50.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:07:50.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:07:50.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:07:50.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:07:51.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:07:51.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:07:51.789 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:07:51.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:07:51.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:07:51.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:07:52.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:07:53.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:07:53.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:07:53.116 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:07:53.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:07:53.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:07:53.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:07:53.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:07:54.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:07:54.444 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:07:54.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:07:54.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:07:54.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:07:54.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:07:54.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:07:55.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:07:55.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:07:55.802 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:07:55.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:07:55.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:07:55.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:07:56.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:07:57.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:07:57.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:07:57.134 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:07:57.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:07:57.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:07:57.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:07:57.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:07:58.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:07:58.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:07:58.643 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:07:58.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:07:58.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:07:58.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:07:59.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:00.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:00.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:00.428 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:08:00.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:08:00.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:00.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:08:00.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:01.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:01.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:01.956 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:08:01.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:08:01.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:01.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:08:02.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:03.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:03.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:03.396 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:08:03.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:08:03.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:03.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:08:04.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:05.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:05.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:05.157 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:08:05.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:08:05.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:05.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:08:05.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:06.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:06.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:06.901 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:08:06.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:08:06.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:06.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:08:07.284 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4231975853,v1:127.0.0.1:6803/4231975853] [v2:127.0.0.1:6804/4231975853,v1:127.0.0.1:6805/4231975853] exists,up 346b2060-e421-4790-83a4-b832b5bf4f69 2024-09-13T14:08:07.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:08:07.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:08:07.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:08:07.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-13T14:08:07.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:08:07.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:08:07.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T14:08:07.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:08:07.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:08:07.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:08:07.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:08:07.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:08:07.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:08:07.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:08:07.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:08:07.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:08:07.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:08:07.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:08:07.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:08:07.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:08:07.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:08:07.288 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:08:07.288 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:08:07.288 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:08:07.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:08:07.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:08:07.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:08:07.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:08:07.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:08:07.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:08:07.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:08:07.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:08:07.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:08:07.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:08:07.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:08:07.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:08:07.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-13T14:08:07.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:08:07.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:08:07.291 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 7e2168cd-b766-4340-825a-272b1b82e9f8 2024-09-13T14:08:07.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7e2168cd-b766-4340-825a-272b1b82e9f8 2024-09-13T14:08:07.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7e2168cd-b766-4340-825a-272b1b82e9f8' 2024-09-13T14:08:07.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:08:07.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBHR+Rm3OrmEhAAYNoPfO7WwA0eU2+RNOBO8Q== 2024-09-13T14:08:07.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBHR+Rm3OrmEhAAYNoPfO7WwA0eU2+RNOBO8Q=="}' 2024-09-13T14:08:07.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7e2168cd-b766-4340-825a-272b1b82e9f8 -i td/osd-scrub-repair/1/new.json 2024-09-13T14:08:07.701 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:08:07.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T14:08:07.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQBHR+Rm3OrmEhAAYNoPfO7WwA0eU2+RNOBO8Q== --osd-uuid 7e2168cd-b766-4340-825a-272b1b82e9f8 2024-09-13T14:08:07.758 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:08:07.757+0000 7f127a78b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:08:07.777 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:08:07.777+0000 7f127a78b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:08:07.778 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:08:07.777+0000 7f127a78b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:08:07.779 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:08:07.781+0000 7f127a78b7c0 -1 bdev(0x56243358bc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:08:07.779 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:08:07.781+0000 7f127a78b7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T14:08:14.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T14:08:14.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:08:14.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T14:08:14.232 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:08:14.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:08:14.633 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:08:14.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T14:08:14.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:08:14.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-13T14:08:14.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:08:14.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:08:14.668 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:08:14.669+0000 7f6d638547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:08:14.698 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:08:14.697+0000 7f6d638547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:08:14.700 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:08:14.701+0000 7f6d638547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:08:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T14:08:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:08:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:08:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:08:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:08:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:14.976 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:08:14.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:08:14.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:14.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:15.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:15.986 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:08:15.985+0000 7f6d638547c0 -1 Falling back to public interface 2024-09-13T14:08:16.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:16.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:16.281 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:08:16.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:08:16.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:16.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:16.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:17.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:17.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:17.618 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:08:17.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:08:17.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:17.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:17.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:18.698 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:08:18.697+0000 7f6d638547c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:08:18.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:18.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:18.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:08:18.936 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:08:18.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:18.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:19.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:20.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:20.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:20.396 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:08:20.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:08:20.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:20.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:20.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:21.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:21.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:21.730 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:08:21.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:08:21.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:21.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:22.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:23.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:23.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:23.048 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:08:23.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:08:23.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:23.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:23.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:24.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:24.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:24.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:08:24.471 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:08:24.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:24.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:24.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:25.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:25.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:25.832 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:08:25.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:08:25.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:25.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:26.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:27.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:27.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:27.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:08:27.173 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:08:27.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:27.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:27.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:28.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:28.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:28.499 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:08:28.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:08:28.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:28.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:28.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:29.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:29.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:29.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:08:29.958 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:08:29.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:29.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:30.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:31.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:31.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:31.296 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:08:31.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:08:31.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:31.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:31.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:32.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:32.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:32.902 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:08:32.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:08:32.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:32.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:33.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:34.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:34.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:34.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:08:34.611 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:08:34.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:34.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:35.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:36.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:36.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:36.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:08:36.743 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:08:36.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:36.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:37.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:38.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:38.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:38.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:08:38.755 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:08:38.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:38.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:39.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:40.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:40.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:40.567 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:08:40.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:08:40.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:40.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:40.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:41.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:41.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:41.966 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:08:41.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:08:41.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:41.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:08:42.300 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/786337955,v1:127.0.0.1:6811/786337955] [v2:127.0.0.1:6812/786337955,v1:127.0.0.1:6813/786337955] exists,up 7e2168cd-b766-4340-825a-272b1b82e9f8 2024-09-13T14:08:42.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:08:42.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:08:42.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:08:42.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-09-13T14:08:42.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T14:08:42.679 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' does not exist 2024-09-13T14:08:42.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T14:08:42.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T14:08:43.278 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T14:08:43.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:08:44.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T14:08:46.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-09-13T14:08:46.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:08:46.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:08:46.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:08:46.123 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:08:46.123 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:08:46.124 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:08:46.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:08:46.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:08:46.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:08:46.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:08:46.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:08:46.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:08:46.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:08:46.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:08:46.247 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:08:46.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:08:46.623 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T14:08:46.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:08:46.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:08:46.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:08:46.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836493 2024-09-13T14:08:46.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836493 2024-09-13T14:08:46.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836493' 2024-09-13T14:08:46.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:08:46.780 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:08:46.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-13T14:08:46.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-13T14:08:46.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836493 1-42949672965' 2024-09-13T14:08:46.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:08:46.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836493 2024-09-13T14:08:46.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:08:46.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:08:46.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836493 2024-09-13T14:08:46.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:08:46.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836493 2024-09-13T14:08:46.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-09-13T14:08:46.934 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836493 2024-09-13T14:08:46.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:08:47.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836493 2024-09-13T14:08:47.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:08:48.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:08:48.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:08:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836493 2024-09-13T14:08:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:08:48.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-13T14:08:48.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:08:48.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:08:48.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-13T14:08:48.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:08:48.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-13T14:08:48.602 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672965 2024-09-13T14:08:48.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-13T14:08:48.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:08:48.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672966 -lt 42949672965 2024-09-13T14:08:48.941 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:08:48.942 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:08:48.942 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:08:49.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T14:08:49.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:08:49.370 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:08:49.370 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:08:49.370 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:08:49.370 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:08:49.370 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:08:49.370 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:08:49.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:08:49.694 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:08:49.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:08:49.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:08:50.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T14:08:50.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:08:50.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:08:50.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-09-13T14:08:50.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T14:08:50.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-09-13T14:08:50.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-13T14:08:50.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T14:08:50.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T14:08:50.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T14:08:50.482 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T14:08:50.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T14:08:50.909 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T14:08:50.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T14:08:50.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T14:08:50.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-13T14:08:51.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-09-13T14:08:51.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2024-09-13T14:08:51.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:08:51.185 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2024-09-13T14:08:51.186 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-13T14:08:51.186 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:08:51.187 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-13T14:08:51.187 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:08:51.515 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T14:08:51.516 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-09-13T14:08:51.516 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T14:08:51.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-09-13T14:08:51.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-13T14:08:51.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-09-13T14:08:51.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-09-13T14:08:51.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T14:08:51.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:08:51.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:08:51.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T14:08:51.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:08:51.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T14:08:51.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:08:51.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:08:51.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T14:08:51.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:08:51.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T14:08:51.846 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:08:51.846 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:08:51.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:08:51.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:08:51.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:08:52.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:08:52.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T14:08:52.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:08:52.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:08:52.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T14:08:52.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:08:52.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:08:52.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-13T14:08:53.950 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-13T14:08:54.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T14:08:54.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:08:54.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:08:54.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T14:08:54.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:08:54.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:08:54.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:08:54.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:08:54.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:08:54.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:08:54.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:08:54.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:08:54.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:08:54.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:08:54.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:08:54.494 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:08:54.494 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:08:54.494 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:08:54.494 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:08:54.495 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:08:54.495 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:08:54.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:08:54.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:08:54.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:08:54.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:08:54.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:08:54.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:08:54.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:08:54.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:08:54.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:08:54.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:08:54.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:08:54.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:08:54.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T14:08:54.497 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T14:08:54.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:08:54.497 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T14:08:54.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T14:08:54.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:08:54.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:08:54.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:08:54.532 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:08:54.533+0000 7fdc0e2a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:08:54.534 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:08:54.533+0000 7fdc0e2a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:08:54.536 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:08:54.537+0000 7fdc0e2a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:08:54.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T14:08:54.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:08:54.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:08:54.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:08:54.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:08:54.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:54.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:08:54.831 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:08:54.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:54.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:08:55.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:55.817 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:08:55.817+0000 7fdc0e2a97c0 -1 Falling back to public interface 2024-09-13T14:08:56.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:56.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:56.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:08:56.149 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:08:56.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:56.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:08:56.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:57.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:57.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:57.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:08:57.478 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:08:57.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:57.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:08:57.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:08:58.544 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:08:58.544+0000 7fdc0e2a97c0 -1 osd.0 21 log_to_monitors true 2024-09-13T14:08:58.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:08:58.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:08:58.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:08:58.802 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:08:58.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:08:58.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:08:59.142 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 26 up_thru 12 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3294690064,v1:127.0.0.1:6803/3294690064] [v2:127.0.0.1:6804/3294690064,v1:127.0.0.1:6805/3294690064] exists,up 346b2060-e421-4790-83a4-b832b5bf4f69 2024-09-13T14:08:59.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:08:59.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:08:59.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:08:59.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:08:59.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:08:59.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:08:59.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:08:59.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:08:59.144 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:08:59.144 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:08:59.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:08:59.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:08:59.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:08:59.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:08:59.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:08:59.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:08:59.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:08:59.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:08:59.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:08:59.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:08:59.645 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T14:08:59.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:08:59.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:08:59.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:08:59.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149698 2024-09-13T14:08:59.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149698 2024-09-13T14:08:59.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698' 2024-09-13T14:08:59.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:08:59.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:08:59.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672969 2024-09-13T14:08:59.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672969 2024-09-13T14:08:59.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698 1-42949672969' 2024-09-13T14:08:59.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:08:59.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149698 2024-09-13T14:08:59.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:08:59.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:08:59.996 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149698 2024-09-13T14:08:59.996 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:08:59.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149698 2024-09-13T14:08:59.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-09-13T14:08:59.997 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 111669149698 2024-09-13T14:08:59.998 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:09:00.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 111669149698 2024-09-13T14:09:00.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:09:01.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:09:01.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:09:01.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149698 -lt 111669149698 2024-09-13T14:09:01.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:09:01.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672969 2024-09-13T14:09:01.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:09:01.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:09:01.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:09:01.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672969 2024-09-13T14:09:01.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672969 2024-09-13T14:09:01.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-13T14:09:01.647 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672969 2024-09-13T14:09:01.647 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:09:01.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672969 -lt 42949672969 2024-09-13T14:09:01.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:09:01.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:09:01.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:09:02.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T14:09:02.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:09:02.382 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:09:02.382 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:09:02.382 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:09:02.382 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:09:02.382 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:09:02.382 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:09:02.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:09:02.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:09:02.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:09:02.734 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:09:03.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T14:09:03.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:09:03.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:09:03.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-09-13T14:09:03.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-13T14:09:03.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:09:03.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:09:03.141 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-13T14:09:03.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-09-13T14:09:03.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-09-13T14:09:03.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.3 2024-09-13T14:09:03.469 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.3 2024-09-13T14:09:03.469 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T14:09:03.469 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:09:03.469 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:09:03.469 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T14:09:03.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:08:43.148878+0000 2024-09-13T14:09:03.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.3 2024-09-13T14:09:04.117 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.3 on osd.1 to repair 2024-09-13T14:09:04.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.3 2024-09-13T14:08:43.148878+0000 2024-09-13T14:09:04.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.3 2024-09-13T14:09:04.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:08:43.148878+0000 2024-09-13T14:09:04.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:09:04.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:09:04.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:09:04.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-13T14:09:04.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T14:09:04.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:09:04.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:09:04.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T14:09:04.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:08:43.148878+0000 '>' 2024-09-13T14:08:43.148878+0000 2024-09-13T14:09:04.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:09:05.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:09:05.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:09:05.433 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-13T14:09:05.433 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T14:09:05.433 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:09:05.434 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:09:05.434 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T14:09:05.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:08:43.148878+0000 '>' 2024-09-13T14:08:43.148878+0000 2024-09-13T14:09:05.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:09:06.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:09:06.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:09:06.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-13T14:09:06.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T14:09:06.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:09:06.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:09:06.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T14:09:07.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:08:43.148878+0000 '>' 2024-09-13T14:08:43.148878+0000 2024-09-13T14:09:07.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:09:08.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:09:08.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:09:08.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-13T14:09:08.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T14:09:08.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:09:08.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:09:08.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T14:09:08.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:09:04.418204+0000 '>' 2024-09-13T14:08:43.148878+0000 2024-09-13T14:09:08.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:09:08.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:09:10.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:09:10.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:09:10.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:09:10.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:09:10.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:09:10.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:09:10.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:09:10.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:09:10.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:09:10.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:09:10.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:09:10.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:09:10.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:09:10.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:09:10.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:09:10.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:09:10.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:09:10.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:09:10.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:09:10.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:09:10.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:09:10.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:09:10.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:09:10.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-13T14:09:11.247 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-13T14:09:11.247 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:09:11.247 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:09:11.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T14:09:11.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:09:11.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:09:11.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:09:11.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:09:11.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:09:11.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:09:11.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:09:11.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:09:11.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:09:11.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:09:11.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:09:11.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:09:11.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:09:11.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:09:11.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:09:11.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:09:11.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:09:11.537 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:09:11.537 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:09:11.537 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:09:11.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:09:11.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:09:11.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:09:11.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:09:11.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:09:11.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:09:11.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:09:11.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:09:11.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:09:11.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:09:11.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:09:11.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:09:11.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:09:11.540 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T14:09:11.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:09:11.540 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:09:11.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:09:11.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:09:11.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:09:11.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:09:11.573 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:11.572+0000 7fa52a1a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:09:11.576 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:11.576+0000 7fa52a1a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:09:11.577 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:11.576+0000 7fa52a1a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:09:11.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:09:11.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:09:11.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:09:11.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:09:11.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:09:11.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:09:11.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:09:11.883 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:09:11.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:09:11.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:09:12.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:09:13.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:09:13.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:09:13.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:09:13.203 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:09:13.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:09:13.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:09:13.293 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:13.292+0000 7fa52a1a17c0 -1 Falling back to public interface 2024-09-13T14:09:13.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:09:14.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:09:14.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:09:14.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:09:14.520 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:09:14.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:09:14.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:09:14.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:09:15.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:09:15.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:09:15.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:09:15.876 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:09:15.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:09:15.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:09:16.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:09:16.515 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:16.520+0000 7fa52a1a17c0 -1 osd.1 28 log_to_monitors true 2024-09-13T14:09:17.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:09:17.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:09:17.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:09:17.215 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:09:17.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:09:17.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:09:17.603 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/848979584,v1:127.0.0.1:6811/848979584] [v2:127.0.0.1:6812/848979584,v1:127.0.0.1:6813/848979584] exists,up 7e2168cd-b766-4340-825a-272b1b82e9f8 2024-09-13T14:09:17.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:09:17.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:09:17.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:09:17.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:09:17.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:09:17.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:09:17.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:09:17.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:09:17.605 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:09:17.605 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:09:17.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:09:17.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:09:17.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:09:17.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:09:17.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:09:17.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:09:17.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:09:17.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:09:17.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:09:18.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:09:18.107 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T14:09:18.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:09:18.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:09:18.107 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:09:18.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149702 2024-09-13T14:09:18.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149702 2024-09-13T14:09:18.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149702' 2024-09-13T14:09:18.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:09:18.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:09:18.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953474 2024-09-13T14:09:18.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953474 2024-09-13T14:09:18.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-09-13T14:09:18.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:09:18.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149702 2024-09-13T14:09:18.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:09:18.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:09:18.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149702 2024-09-13T14:09:18.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:09:18.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149702 2024-09-13T14:09:18.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-09-13T14:09:18.419 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 111669149702 2024-09-13T14:09:18.419 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:09:18.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149702 -lt 111669149702 2024-09-13T14:09:18.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:09:18.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-137438953474 2024-09-13T14:09:18.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:09:18.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:09:18.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:09:18.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-137438953474 2024-09-13T14:09:18.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953474 2024-09-13T14:09:18.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-09-13T14:09:18.731 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 137438953474 2024-09-13T14:09:18.731 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:09:19.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 137438953474 2024-09-13T14:09:19.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:09:20.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:09:20.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:09:20.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 137438953474 2024-09-13T14:09:20.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:09:21.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T14:09:21.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:09:21.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953474 -lt 137438953474 2024-09-13T14:09:21.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:09:21.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:09:21.728 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:09:22.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T14:09:22.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:09:22.131 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:09:22.131 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:09:22.131 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:09:22.131 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:09:22.131 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:09:22.131 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:09:22.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:09:22.464 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:09:22.464 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:09:22.464 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:09:22.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T14:09:22.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:09:22.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:09:22.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:09:22.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:09:22.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T14:09:22.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:09:22.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:09:22.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:09:22.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:09:22.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:09:22.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:09:22.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:09:22.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:09:23.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:09:23.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:09:23.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:09:23.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:09:23.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:09:23.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:09:23.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:09:23.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:09:23.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:09:23.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:09:23.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:09:23.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:09:23.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:09:23.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:09:23.036 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:09:23.036 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:09:23.036 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:09:23.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:09:23.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:09:23.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:09:23.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T14:09:23.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T14:09:23.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T14:09:23.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T14:09:23.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:09:23.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:09:23.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:09:23.039 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:09:23.039 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:09:23.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:09:23.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:09:23.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:09:23.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:09:23.040 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:09:23.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:09:23.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:09:23.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:09:23.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:09:23.042 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:09:23.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:09:23.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:09:23.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:09:23.044 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:09:23.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:09:23.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:09:23.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:09:23.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:09:23.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:09:23.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:09:23.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:09:23.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:09:23.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:09:23.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T14:09:23.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:09:23.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:09:23.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:09:23.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T14:09:23.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:09:23.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T14:09:23.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:09:23.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T14:09:23.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-09-13T14:09:23.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-09-13T14:09:23.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-09-13T14:09:23.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-09-13T14:09:23.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-09-13T14:09:23.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-09-13T14:09:23.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-09-13T14:09:23.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2024-09-13T14:09:23.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2024-09-13T14:09:23.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=1 2024-09-13T14:09:23.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:234: standard_scrub_cluster: local poolname=testpool 2024-09-13T14:09:23.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: args['pool_name']=testpool 2024-09-13T14:09:23.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-09-13T14:09:23.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:237: standard_scrub_cluster: local debug_msg=dbg 2024-09-13T14:09:23.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:240: standard_scrub_cluster: local saved_echo_flag=x 2024-09-13T14:09:23.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: set +x 2024-09-13T14:09:24.815 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 d30fccd7-8f45-4f7f-bb3c-6ab91be7ea2e 2024-09-13T14:09:25.067 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:09:25.119 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:25.124+0000 7f830d7a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:09:25.121 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:25.124+0000 7f830d7a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:09:25.122 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:25.124+0000 7f830d7a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:09:25.122 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:25.128+0000 7f830d7a57c0 -1 bdev(0x56369fa5fc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:09:25.122 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:25.128+0000 7f830d7a57c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T14:09:31.325 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:09:31.750 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:09:31.784 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:31.788+0000 7fb87bde67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:09:31.835 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:31.840+0000 7fb87bde67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:09:31.836 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:31.840+0000 7fb87bde67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:09:32.067 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:09:33.361 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:33.364+0000 7fb87bde67c0 -1 Falling back to public interface 2024-09-13T14:09:33.412 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:09:34.708 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:09:36.047 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:09:36.168 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:36.172+0000 7fb87bde67c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:09:37.431 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:09:38.768 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:09:40.147 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:09:41.482 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:09:42.903 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:09:44.246 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:09:45.567 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:09:46.927 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:09:48.267 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:09:49.817 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:09:51.538 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:09:53.534 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:09:54.952 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:09:56.682 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:09:58.419 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:09:58.770 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3313785230,v1:127.0.0.1:6803/3313785230] [v2:127.0.0.1:6804/3313785230,v1:127.0.0.1:6805/3313785230] exists,up d30fccd7-8f45-4f7f-bb3c-6ab91be7ea2e 2024-09-13T14:09:58.777 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 c7199836-2bab-4031-91db-2a10a9b56fd9 2024-09-13T14:09:59.194 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:09:59.258 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:59.259+0000 7f94a1dbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:09:59.279 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:59.283+0000 7f94a1dbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:09:59.280 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:59.283+0000 7f94a1dbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:09:59.281 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:59.283+0000 7f94a1dbe7c0 -1 bdev(0x558d4fe34800 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:09:59.281 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:09:59.283+0000 7f94a1dbe7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T14:10:05.770 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:10:06.167 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:10:06.206 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:10:06.211+0000 7fdbf6e887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:10:06.228 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:10:06.231+0000 7fdbf6e887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:10:06.229 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:10:06.231+0000 7fdbf6e887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:10:06.505 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:10:07.257 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:10:07.259+0000 7fdbf6e887c0 -1 Falling back to public interface 2024-09-13T14:10:07.838 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:10:09.175 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:10:10.111 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:10:10.115+0000 7fdbf6e887c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:10:10.550 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:10:12.072 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:10:13.468 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:10:14.927 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:10:16.263 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:10:17.592 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:10:18.994 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:10:20.329 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:10:21.680 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:10:23.200 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:10:24.823 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:10:26.419 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:10:27.939 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:10:29.295 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:10:30.732 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:10:32.305 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:10:33.745 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:10:34.074 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3656012845,v1:127.0.0.1:6811/3656012845] [v2:127.0.0.1:6812/3656012845,v1:127.0.0.1:6813/3656012845] exists,up c7199836-2bab-4031-91db-2a10a9b56fd9 2024-09-13T14:10:34.079 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 41dad250-ddbc-4987-b408-2a3f087c54ff 2024-09-13T14:10:34.466 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:10:34.524 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:10:34.527+0000 7f0c9686c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:10:34.543 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:10:34.547+0000 7f0c9686c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:10:34.544 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:10:34.547+0000 7f0c9686c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:10:34.544 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:10:34.547+0000 7f0c9686c7c0 -1 bdev(0x5572c1957c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T14:10:34.544 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:10:34.547+0000 7f0c9686c7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T14:10:40.980 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T14:10:41.385 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T14:10:41.423 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:10:41.426+0000 7ff2248ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:10:41.448 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:10:41.450+0000 7ff2248ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:10:41.450 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:10:41.454+0000 7ff2248ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:10:41.761 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:10:42.744 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:10:42.746+0000 7ff2248ce7c0 -1 Falling back to public interface 2024-09-13T14:10:43.108 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:10:44.445 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:10:45.430 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:10:45.434+0000 7ff2248ce7c0 -1 osd.2 0 log_to_monitors true 2024-09-13T14:10:45.771 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:10:47.099 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:10:48.473 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:10:49.798 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:10:51.220 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:10:52.590 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:10:53.913 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:10:55.487 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:10:56.816 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:10:58.178 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:10:59.568 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:11:01.241 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:11:02.767 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:11:04.481 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:11:06.073 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:11:07.561 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:11:07.881 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4290038563,v1:127.0.0.1:6819/4290038563] [v2:127.0.0.1:6820/4290038563,v1:127.0.0.1:6821/4290038563] exists,up 41dad250-ddbc-4987-b408-2a3f087c54ff 2024-09-13T14:11:08.291 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'testpool' created 2024-09-13T14:11:10.404 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836505 2024-09-13T14:11:12.062 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 38654705678 2024-09-13T14:11:12.429 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 60129542151 2024-09-13T14:11:15.245 INFO:tasks.workunit.client.0.smithi112.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-09-13T14:11:15.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-09-13T14:11:15.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-09-13T14:11:15.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-09-13T14:11:15.808 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 1 size to 2 2024-09-13T14:11:15.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-09-13T14:11:15.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:11:15.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:11:15.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:11:15.831 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:11:15.831 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:11:15.832 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:11:15.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:11:15.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:11:15.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:11:15.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:11:15.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:11:15.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:11:15.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:11:15.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:11:15.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:11:16.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:11:16.434 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:11:16.434 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:11:16.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:11:16.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:11:16.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:11:16.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-13T14:11:16.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-13T14:11:16.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-13T14:11:16.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:11:16.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:11:16.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705680 2024-09-13T14:11:16.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705680 2024-09-13T14:11:16.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-38654705680' 2024-09-13T14:11:16.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:11:16.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:11:16.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=60129542153 2024-09-13T14:11:16.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 60129542153 2024-09-13T14:11:16.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-38654705680 2-60129542153' 2024-09-13T14:11:16.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:11:16.866 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-13T14:11:16.866 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:11:16.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:11:16.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-13T14:11:16.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:11:16.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-13T14:11:16.869 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836507 2024-09-13T14:11:16.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-13T14:11:16.869 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:11:17.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836507 2024-09-13T14:11:17.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:11:18.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:11:18.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:11:18.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-13T14:11:18.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:11:18.510 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705680 2024-09-13T14:11:18.510 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:11:18.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:11:18.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705680 2024-09-13T14:11:18.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:11:18.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705680 2024-09-13T14:11:18.515 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 38654705680 2024-09-13T14:11:18.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705680' 2024-09-13T14:11:18.515 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:11:18.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705680 -lt 38654705680 2024-09-13T14:11:18.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:11:18.847 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-60129542153 2024-09-13T14:11:18.847 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:11:18.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:11:18.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-60129542153 2024-09-13T14:11:18.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:11:18.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=60129542153 2024-09-13T14:11:18.850 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 60129542153 2024-09-13T14:11:18.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2024-09-13T14:11:18.850 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:11:19.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 60129542153 -lt 60129542153 2024-09-13T14:11:19.162 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:11:19.162 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:11:19.162 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:11:19.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:11:19.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:11:19.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:11:19.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:11:19.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:11:19.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:11:19.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:11:19.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:11:19.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:11:19.872 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:11:19.872 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:11:19.872 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:11:20.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:11:20.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:11:20.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:11:20.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-09-13T14:11:20.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-09-13T14:11:20.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-13T14:11:20.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-09-13T14:11:20.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-13T14:11:20.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:11:20.408 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-13T14:11:20.408 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:11:20.409 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:11:20.409 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:11:20.409 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:11:20.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T14:11:20.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:11:20.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T14:11:21.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T14:11:21.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:11:21.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:11:21.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T14:11:21.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:11:21.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T14:11:21.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:11:21.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:11:21.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T14:11:21.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:11:21.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T14:11:21.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:11:21.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:11:21.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:11:21.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:11:21.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:11:21.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:11:21.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T14:11:21.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:11:21.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:11:21.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T14:11:21.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:11:21.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:11:21.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-13T14:11:22.991 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-13T14:11:23.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T14:11:23.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:11:23.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:11:23.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T14:11:23.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:11:23.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:11:23.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:11:23.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:11:23.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:11:23.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:11:23.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:11:23.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:11:23.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:11:23.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:11:23.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:11:23.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:11:23.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:11:23.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:11:23.528 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:11:23.528 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:11:23.528 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:11:23.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:11:23.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:11:23.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:11:23.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:11:23.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:11:23.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:11:23.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:11:23.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:11:23.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:11:23.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:11:23.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:11:23.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:11:23.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T14:11:23.530 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T14:11:23.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:11:23.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T14:11:23.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T14:11:23.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:11:23.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:11:23.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:11:23.569 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:11:23.569+0000 7ff528d857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:11:23.572 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:11:23.573+0000 7ff528d857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:11:23.573 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:11:23.573+0000 7ff528d857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:11:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T14:11:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:11:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:11:23.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:11:23.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:11:23.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:11:23.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:11:23.868 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:11:23.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:11:23.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:11:24.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:11:24.605 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:11:24.605+0000 7ff528d857c0 -1 Falling back to public interface 2024-09-13T14:11:25.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:11:25.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:11:25.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:11:25.212 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:11:25.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:11:25.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:11:25.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:11:26.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:11:26.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:11:26.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:11:26.546 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:11:26.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:11:26.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:11:26.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:11:27.345 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:11:27.345+0000 7ff528d857c0 -1 osd.0 20 log_to_monitors true 2024-09-13T14:11:27.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:11:27.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:11:27.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:11:27.870 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:11:27.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:11:27.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:11:28.199 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/77937997,v1:127.0.0.1:6803/77937997] [v2:127.0.0.1:6804/77937997,v1:127.0.0.1:6805/77937997] exists,up d30fccd7-8f45-4f7f-bb3c-6ab91be7ea2e 2024-09-13T14:11:28.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:11:28.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:11:28.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:11:28.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:11:28.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:11:28.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:11:28.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:11:28.200 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:11:28.202 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:11:28.202 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:11:28.202 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:11:28.202 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:11:28.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:11:28.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:11:28.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:11:28.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:11:28.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:11:28.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:11:28.306 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:11:28.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:11:28.718 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:11:28.718 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:11:28.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:11:28.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:11:28.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:11:28.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215106 2024-09-13T14:11:28.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215106 2024-09-13T14:11:28.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106' 2024-09-13T14:11:28.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:11:28.887 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:11:29.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705684 2024-09-13T14:11:29.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705684 2024-09-13T14:11:29.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106 1-38654705684' 2024-09-13T14:11:29.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:11:29.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:11:29.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=60129542157 2024-09-13T14:11:29.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 60129542157 2024-09-13T14:11:29.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106 1-38654705684 2-60129542157' 2024-09-13T14:11:29.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:11:29.163 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-103079215106 2024-09-13T14:11:29.164 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:11:29.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:11:29.165 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-103079215106 2024-09-13T14:11:29.165 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:11:29.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215106 2024-09-13T14:11:29.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-09-13T14:11:29.166 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 103079215106 2024-09-13T14:11:29.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:11:29.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 103079215106 2024-09-13T14:11:29.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:11:30.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:11:30.472 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:11:30.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215106 -lt 103079215106 2024-09-13T14:11:30.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:11:30.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705684 2024-09-13T14:11:30.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:11:30.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:11:30.796 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705684 2024-09-13T14:11:30.796 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:11:30.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705684 2024-09-13T14:11:30.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705684' 2024-09-13T14:11:30.797 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 38654705684 2024-09-13T14:11:30.797 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:11:31.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705684 -lt 38654705684 2024-09-13T14:11:31.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:11:31.128 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-60129542157 2024-09-13T14:11:31.128 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:11:31.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:11:31.131 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-60129542157 2024-09-13T14:11:31.131 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:11:31.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=60129542157 2024-09-13T14:11:31.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2024-09-13T14:11:31.132 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 60129542157 2024-09-13T14:11:31.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:11:31.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 60129542157 -lt 60129542157 2024-09-13T14:11:31.459 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:11:31.460 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:11:31.460 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:11:31.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:11:31.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:11:31.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:11:31.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:11:31.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:11:31.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:11:31.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:11:31.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:11:32.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:11:32.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:11:32.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:11:32.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:11:32.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:11:32.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:11:32.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:11:32.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-09-13T14:11:32.805 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T14:11:32.805 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-13T14:11:32.805 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:11:32.818 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T14:11:32.818 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-09-13T14:11:32.818 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:11:32.830 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2: { 2024-09-13T14:11:32.830 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-09-13T14:11:32.830 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:11:32.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-09-13T14:11:32.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-13T14:11:32.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:11:32.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2024-09-13T14:11:32.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:11:33.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-09-13T14:11:33.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-09-13T14:11:33.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:11:33.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:11:33.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:11:33.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:11:33.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-09-13T14:11:33.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-09-13T14:11:33.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:11:33.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:11:33.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:11:33.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:11:33.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-09-13T14:11:08.248533+0000 2024-09-13T14:11:33.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-09-13T14:11:33.959 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T14:11:33.959 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T14:11:33.959 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T14:11:33.959 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-08-30T14:09:53.965615+0000" 2024-09-13T14:11:33.959 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:11:33.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-09-13T14:11:08.248533+0000 2024-09-13T14:11:33.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T14:11:33.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:11:08.248533+0000 2024-09-13T14:11:33.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:11:33.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:11:33.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:11:33.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:11:33.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:11:33.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:11:33.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:11:33.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:11:34.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:11:08.248533+0000 '>' 2024-09-13T14:11:08.248533+0000 2024-09-13T14:11:34.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:11:35.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:11:35.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:11:35.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:11:35.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:11:35.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:11:35.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:11:35.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:11:35.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:11:08.248533+0000 '>' 2024-09-13T14:11:08.248533+0000 2024-09-13T14:11:35.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:11:36.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:11:36.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:11:36.647 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:11:36.647 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:11:36.647 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:11:36.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:11:36.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:11:36.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:11:08.248533+0000 '>' 2024-09-13T14:11:08.248533+0000 2024-09-13T14:11:36.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:11:37.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:11:37.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:11:37.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:11:37.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:11:37.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:11:37.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:11:37.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:11:38.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:11:08.248533+0000 '>' 2024-09-13T14:11:08.248533+0000 2024-09-13T14:11:38.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:11:39.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:11:39.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:11:39.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:11:39.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:11:39.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:11:39.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:11:39.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:11:39.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:11:34.455394+0000 '>' 2024-09-13T14:11:08.248533+0000 2024-09-13T14:11:39.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:11:39.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-09-13T14:11:39.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:11:39.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:11:39.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:11:39.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:11:39.627 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:11:39.627 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:11:39.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:11:39.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:11:39.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:11:39.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:11:39.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:11:39.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:11:39.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:11:39.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:11:39.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:11:40.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:11:40.048 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:11:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:11:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:11:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:11:40.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:11:40.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215109 2024-09-13T14:11:40.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215109 2024-09-13T14:11:40.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215109' 2024-09-13T14:11:40.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:11:40.233 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:11:40.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705687 2024-09-13T14:11:40.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705687 2024-09-13T14:11:40.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215109 1-38654705687' 2024-09-13T14:11:40.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:11:40.375 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:11:40.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=60129542160 2024-09-13T14:11:40.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 60129542160 2024-09-13T14:11:40.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215109 1-38654705687 2-60129542160' 2024-09-13T14:11:40.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:11:40.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-103079215109 2024-09-13T14:11:40.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:11:40.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:11:40.518 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-103079215109 2024-09-13T14:11:40.518 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:11:40.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215109 2024-09-13T14:11:40.520 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 103079215109 2024-09-13T14:11:40.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 103079215109' 2024-09-13T14:11:40.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:11:40.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215108 -lt 103079215109 2024-09-13T14:11:40.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:11:41.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:11:41.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:11:42.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215109 -lt 103079215109 2024-09-13T14:11:42.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:11:42.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:11:42.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705687 2024-09-13T14:11:42.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:11:42.186 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705687 2024-09-13T14:11:42.186 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:11:42.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705687 2024-09-13T14:11:42.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705687' 2024-09-13T14:11:42.188 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 38654705687 2024-09-13T14:11:42.188 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:11:42.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705687 -lt 38654705687 2024-09-13T14:11:42.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:11:42.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:11:42.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-60129542160 2024-09-13T14:11:42.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:11:42.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-60129542160 2024-09-13T14:11:42.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:11:42.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=60129542160 2024-09-13T14:11:42.533 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 60129542160 2024-09-13T14:11:42.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2024-09-13T14:11:42.533 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:11:42.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 60129542160 -lt 60129542160 2024-09-13T14:11:42.856 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:11:42.857 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:11:42.857 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:11:43.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:11:43.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:11:43.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:11:43.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:11:43.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:11:43.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:11:43.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:11:43.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:11:43.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:11:43.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:11:43.609 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:11:43.609 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:11:43.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:11:43.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:11:43.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:11:43.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-09-13T14:11:44.301 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T14:11:44.301 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-13T14:11:34.609613+0000 20'1 25:61 [1,0] 1 [1,0] 1 20'1 2024-09-13T14:11:34.455394+0000 20'1 2024-09-13T14:11:34.455394+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:11:34.455394+0000 1 0 2024-09-13T14:11:44.301 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T14:11:44.301 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T14:11:44.301 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:11:44.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-09-13T14:11:44.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-13T14:11:44.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:11:44.317 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-13T14:11:44.317 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:11:44.318 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:11:44.318 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:11:44.318 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:11:44.632 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T14:11:44.632 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:11:44.632 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T14:11:45.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-13T14:11:45.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:11:45.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:11:45.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T14:11:45.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:11:45.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-13T14:11:45.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:11:45.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:11:45.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T14:11:45.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:11:45.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T14:11:45.005 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:11:45.005 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:11:45.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:11:45.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:11:45.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:11:45.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:11:45.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-13T14:11:45.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:11:45.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:11:45.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T14:11:45.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:11:45.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:11:45.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-13T14:11:45.782 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:11:45.782 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:11:46.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T14:11:46.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:11:46.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:11:46.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T14:11:46.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:11:46.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:11:46.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:11:46.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:11:46.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:11:46.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:11:46.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:11:46.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:11:46.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:11:46.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:11:46.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:11:46.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:11:46.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:11:46.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:11:46.090 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:11:46.090 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:11:46.090 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:11:46.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:11:46.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:11:46.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:11:46.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:11:46.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:11:46.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:11:46.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:11:46.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:11:46.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:11:46.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:11:46.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:11:46.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:11:46.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T14:11:46.092 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T14:11:46.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:11:46.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T14:11:46.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T14:11:46.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:11:46.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:11:46.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:11:46.126 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:11:46.125+0000 7fe5aa6877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:11:46.129 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:11:46.129+0000 7fe5aa6877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:11:46.130 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:11:46.133+0000 7fe5aa6877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:11:46.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T14:11:46.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:11:46.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:11:46.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:11:46.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:11:46.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:11:46.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:11:46.411 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:11:46.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:11:46.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:11:46.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:11:47.161 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:11:47.160+0000 7fe5aa6877c0 -1 Falling back to public interface 2024-09-13T14:11:47.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:11:47.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:11:47.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:11:47.720 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:11:47.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:11:47.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:11:48.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:11:49.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:11:49.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:11:49.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:11:49.057 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:11:49.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:11:49.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:11:49.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:11:50.286 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:11:50.288+0000 7fe5aa6877c0 -1 osd.0 25 log_to_monitors true 2024-09-13T14:11:50.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:11:50.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:11:50.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:11:50.383 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:11:50.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:11:50.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:11:50.766 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/4008445234,v1:127.0.0.1:6803/4008445234] [v2:127.0.0.1:6804/4008445234,v1:127.0.0.1:6805/4008445234] exists,up d30fccd7-8f45-4f7f-bb3c-6ab91be7ea2e 2024-09-13T14:11:50.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:11:50.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:11:50.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:11:50.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:11:50.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:11:50.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:11:50.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:11:50.766 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:11:50.767 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:11:50.767 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:11:50.767 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:11:50.767 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:11:50.768 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:11:50.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:11:50.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:11:50.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:11:50.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:11:50.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:11:50.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:11:51.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:11:51.298 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:11:51.299 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:11:51.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:11:51.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:11:51.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:11:51.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051586 2024-09-13T14:11:51.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051586 2024-09-13T14:11:51.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586' 2024-09-13T14:11:51.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:11:51.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:11:51.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705690 2024-09-13T14:11:51.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705690 2024-09-13T14:11:51.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586 1-38654705690' 2024-09-13T14:11:51.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:11:51.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:11:51.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=60129542163 2024-09-13T14:11:51.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 60129542163 2024-09-13T14:11:51.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586 1-38654705690 2-60129542163' 2024-09-13T14:11:51.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:11:51.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051586 2024-09-13T14:11:51.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:11:51.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:11:51.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051586 2024-09-13T14:11:51.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:11:51.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051586 2024-09-13T14:11:51.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-09-13T14:11:51.741 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 124554051586 2024-09-13T14:11:51.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:11:52.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-13T14:11:52.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:11:52.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705690 2024-09-13T14:11:52.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:11:52.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:11:52.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705690 2024-09-13T14:11:52.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:11:52.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705690 2024-09-13T14:11:52.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705690' 2024-09-13T14:11:52.048 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 38654705690 2024-09-13T14:11:52.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:11:52.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705689 -lt 38654705690 2024-09-13T14:11:52.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:11:53.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:11:53.359 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:11:53.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705690 -lt 38654705690 2024-09-13T14:11:53.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:11:53.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-60129542163 2024-09-13T14:11:53.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:11:53.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:11:53.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-60129542163 2024-09-13T14:11:53.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:11:53.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=60129542163 2024-09-13T14:11:53.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 60129542163' 2024-09-13T14:11:53.699 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 60129542163 2024-09-13T14:11:53.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:11:54.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 60129542164 -lt 60129542163 2024-09-13T14:11:54.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:11:54.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:11:54.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:11:54.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:11:54.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:11:54.439 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:11:54.439 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:11:54.439 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:11:54.439 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:11:54.439 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:11:54.439 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:11:54.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:11:54.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:11:54.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:11:54.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:11:55.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:11:55.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:11:55.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:11:55.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-09-13T14:11:55.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-13T14:11:55.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:11:55.185 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-13T14:11:55.185 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:11:55.185 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:11:55.185 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:11:55.185 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:11:55.550 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T14:11:55.550 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:11:55.550 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T14:11:55.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-13T14:11:55.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:11:55.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:11:55.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T14:11:55.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:11:55.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-13T14:11:55.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:11:55.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:11:55.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T14:11:55.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:11:55.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T14:11:55.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:11:55.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:11:55.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:11:55.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:11:55.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:11:56.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:11:56.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-13T14:11:56.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:11:56.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:11:56.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T14:11:56.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:11:56.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:11:56.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-13T14:11:57.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T14:11:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:11:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:11:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T14:11:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:11:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:11:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:11:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:11:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:11:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:11:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:11:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:11:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:11:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:11:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:11:57.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:11:57.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:11:57.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:11:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:11:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:11:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:11:57.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:11:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:11:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:11:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:11:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:11:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:11:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:11:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:11:57.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:11:57.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:11:57.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:11:57.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:11:57.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T14:11:57.021 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T14:11:57.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:11:57.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T14:11:57.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T14:11:57.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:11:57.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:11:57.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:11:57.051 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:11:57.052+0000 7fef3f4c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:11:57.054 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:11:57.056+0000 7fef3f4c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:11:57.055 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:11:57.056+0000 7fef3f4c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:11:57.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T14:11:57.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:11:57.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:11:57.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:11:57.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:11:57.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:11:57.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:11:57.348 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:11:57.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:11:57.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:11:57.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:11:58.329 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:11:58.332+0000 7fef3f4c37c0 -1 Falling back to public interface 2024-09-13T14:11:58.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:11:58.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:11:58.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:11:58.687 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:11:58.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:11:58.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:11:59.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:00.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:00.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:00.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:12:00.020 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:12:00.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:00.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:00.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:12:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:12:01.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:01.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:01.507 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:01.512+0000 7fef3f4c37c0 -1 osd.0 30 log_to_monitors true 2024-09-13T14:12:01.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:02.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:02.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:02.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:12:02.745 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:12:02.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:02.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:03.071 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 34 up_thru 0 down_at 31 last_clean_interval [29,30) [v2:127.0.0.1:6802/1840901480,v1:127.0.0.1:6803/1840901480] [v2:127.0.0.1:6804/1840901480,v1:127.0.0.1:6805/1840901480] exists,up d30fccd7-8f45-4f7f-bb3c-6ab91be7ea2e 2024-09-13T14:12:03.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:12:03.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:12:03.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:12:03.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:12:03.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:12:03.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:12:03.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:12:03.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:12:03.072 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:12:03.073 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:12:03.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:12:03.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:12:03.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:12:03.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:12:03.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:12:03.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:12:03.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:12:03.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:12:03.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:12:03.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:12:03.477 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:12:03.477 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:12:03.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:12:03.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:12:03.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:12:03.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888066 2024-09-13T14:12:03.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888066 2024-09-13T14:12:03.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066' 2024-09-13T14:12:03.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:12:03.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:12:03.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705694 2024-09-13T14:12:03.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705694 2024-09-13T14:12:03.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066 1-38654705694' 2024-09-13T14:12:03.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:12:03.767 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:12:03.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=60129542167 2024-09-13T14:12:03.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 60129542167 2024-09-13T14:12:03.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-146028888066 1-38654705694 2-60129542167' 2024-09-13T14:12:03.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:12:03.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:12:03.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-146028888066 2024-09-13T14:12:03.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:12:03.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-146028888066 2024-09-13T14:12:03.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:12:03.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888066 2024-09-13T14:12:03.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-09-13T14:12:03.912 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 146028888066 2024-09-13T14:12:03.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:12:04.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 146028888066 2024-09-13T14:12:04.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:12:05.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:12:05.232 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:12:05.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 146028888066 2024-09-13T14:12:05.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:12:06.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T14:12:06.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:12:06.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888066 -lt 146028888066 2024-09-13T14:12:06.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:12:06.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705694 2024-09-13T14:12:06.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:12:06.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:12:06.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:12:06.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705694 2024-09-13T14:12:06.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705694 2024-09-13T14:12:06.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705694' 2024-09-13T14:12:06.916 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 38654705694 2024-09-13T14:12:06.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:12:07.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705694 -lt 38654705694 2024-09-13T14:12:07.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:12:07.253 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-60129542167 2024-09-13T14:12:07.253 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:12:07.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:12:07.254 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-60129542167 2024-09-13T14:12:07.254 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:12:07.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=60129542167 2024-09-13T14:12:07.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 60129542167' 2024-09-13T14:12:07.257 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 60129542167 2024-09-13T14:12:07.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:12:07.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 60129542167 -lt 60129542167 2024-09-13T14:12:07.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:12:07.583 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:12:07.583 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:12:07.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:12:07.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:12:07.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:12:07.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:12:07.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:12:07.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:12:07.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:12:07.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:12:08.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:12:08.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:12:08.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:12:08.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:12:08.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:12:08.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:12:08.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:12:08.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:12:08.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-13T14:12:08.703 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:11:34.449+0000 7fdbdae12640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-13T14:12:08.704 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:11:34.449+0000 7fdbdae12640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-13T14:12:08.704 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:11:34.449+0000 7fdbdae12640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-13T14:12:08.704 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:11:34.449+0000 7fdbdae12640 20 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-09-13T14:12:08.704 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:11:34.449+0000 7fdbdae12640 19 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-13T14:12:08.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T14:12:08.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:12:08.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:12:08.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:12:08.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:12:08.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:12:08.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:12:08.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:12:08.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:12:08.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:12:08.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:12:08.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:12:08.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:12:08.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:12:08.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:12:08.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:12:08.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:12:08.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:12:08.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:12:08.820 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:12:08.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:12:08.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:12:08.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:12:08.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:12:08.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:12:08.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:12:08.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:12:08.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:12:08.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:12:08.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T14:12:08.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T14:12:08.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T14:12:08.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T14:12:08.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:12:08.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:12:08.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:12:08.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:12:08.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:12:08.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:12:08.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:12:08.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:12:08.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:12:08.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:12:08.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:12:08.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:12:08.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:12:08.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:12:08.845 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:12:08.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:12:08.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:12:08.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:12:08.846 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:12:08.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:12:08.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:12:08.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:12:08.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:12:08.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:12:08.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:12:08.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:12:08.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:12:08.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:12:08.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T14:12:08.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:12:08.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:12:08.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:12:08.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T14:12:08.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:12:08.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T14:12:08.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:12:08.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T14:12:08.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-09-13T14:12:08.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-09-13T14:12:08.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-09-13T14:12:08.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-09-13T14:12:08.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T14:12:08.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T14:12:08.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T14:12:08.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T14:12:08.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T14:12:08.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T14:12:09.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T14:12:09.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:12:09.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:12:09.322 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:12:09.322 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:12:09.322 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:12:09.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:12:09.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T14:12:09.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T14:12:09.516 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T14:12:09.516 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:12:09.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:12:09.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T14:12:09.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T14:12:09.517 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:12:09.517 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:12:09.517 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:12:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:12:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:12:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:12:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:12:09.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:12:09.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T14:12:09.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T14:12:09.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:12:09.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:12:09.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T14:12:09.631 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:12:09.631 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:12:09.631 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T14:12:09.631 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:12:09.631 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:12:09.632 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:12:09.632 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:12:09.632 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:12:09.632 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:12:09.632 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T14:12:09.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-09-13T14:12:09.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T14:12:09.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T14:12:09.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T14:12:09.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T14:12:09.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T14:12:09.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T14:12:10.596 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T14:12:10.596 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:12:10.596 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:12:10.597 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:12:10.597 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:12:10.597 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:12:10.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:12:10.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:12:10.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:12:10.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:12:10.631 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-09-13T14:12:10.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-09-13T14:12:10.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:12:10.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:12:10.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:12:10.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T14:12:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:12:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:12:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:12:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:12:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:12:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:12:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:12:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:12:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:12:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:12:10.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:12:10.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:12:10.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:12:10.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:12:10.636 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:12:10.636 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:12:10.636 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:12:10.636 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:12:10.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:12:10.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:12:10.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:12:10.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:12:10.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:12:10.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:12:10.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:12:10.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:12:10.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:12:10.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:12:10.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:12:10.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:12:10.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:12:10.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:12:10.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=846d515c-af3e-42ac-9df5-e0a3f3ad9bc8 2024-09-13T14:12:10.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 846d515c-af3e-42ac-9df5-e0a3f3ad9bc8' 2024-09-13T14:12:10.638 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 846d515c-af3e-42ac-9df5-e0a3f3ad9bc8 2024-09-13T14:12:10.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:12:10.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA6SORmX7tVJxAAmbzoMLQ3s2ysgBO/+u+86w== 2024-09-13T14:12:10.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA6SORmX7tVJxAAmbzoMLQ3s2ysgBO/+u+86w=="}' 2024-09-13T14:12:10.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 846d515c-af3e-42ac-9df5-e0a3f3ad9bc8 -i td/osd-scrub-repair/0/new.json 2024-09-13T14:12:10.885 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:12:10.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T14:12:10.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA6SORmX7tVJxAAmbzoMLQ3s2ysgBO/+u+86w== --osd-uuid 846d515c-af3e-42ac-9df5-e0a3f3ad9bc8 2024-09-13T14:12:10.945 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:10.948+0000 7f233ec447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:12:10.947 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:10.952+0000 7f233ec447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:12:10.949 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:10.952+0000 7f233ec447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:12:10.949 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:10.952+0000 7f233ec447c0 -1 bdev(0x55e2dac61c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:12:10.949 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:10.952+0000 7f233ec447c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T14:12:17.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T14:12:17.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:12:17.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T14:12:17.139 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:12:17.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:12:17.548 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:12:17.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T14:12:17.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:12:17.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:12:17.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:12:17.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:12:17.583 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:17.588+0000 7f12f55cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:12:17.586 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:17.588+0000 7f12f55cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:12:17.587 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:17.592+0000 7f12f55cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:12:17.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T14:12:17.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:12:17.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:12:17.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:12:17.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:12:17.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:17.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:12:17.851 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:12:17.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:17.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:18.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:19.109 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:19.112+0000 7f12f55cf7c0 -1 Falling back to public interface 2024-09-13T14:12:19.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:19.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:19.190 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:12:19.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:12:19.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:19.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:19.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:20.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:20.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:20.506 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:12:20.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:12:20.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:20.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:20.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:21.756 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:21.759+0000 7f12f55cf7c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:12:21.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:21.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:21.823 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:12:21.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:12:21.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:21.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:22.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:23.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:23.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:23.222 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:12:23.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:12:23.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:23.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:23.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:24.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:24.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:24.706 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:12:24.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:12:24.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:24.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:25.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:26.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:26.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:26.032 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:12:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:12:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:26.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:27.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:27.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:27.352 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:12:27.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:12:27.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:27.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:27.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:28.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:28.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:28.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:12:28.755 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:12:28.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:28.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:29.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:30.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:30.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:30.080 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:12:30.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:12:30.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:30.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:30.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:31.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:31.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:31.491 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:12:31.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:12:31.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:31.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:31.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:32.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:32.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:32.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:12:32.820 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:12:32.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:32.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:33.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:34.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:34.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:34.155 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:12:34.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:12:34.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:34.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:34.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:35.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:35.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:35.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:12:35.866 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:12:35.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:35.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:36.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:37.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:37.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:37.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:12:37.603 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:12:37.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:37.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:38.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:39.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:39.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:39.587 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:12:39.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:12:39.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:39.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:40.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:41.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:41.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:41.271 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:12:41.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:12:41.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:41.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:42.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:43.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:43.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:43.138 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:12:43.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:12:43.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:43.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:12:43.520 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/288278290,v1:127.0.0.1:6803/288278290] [v2:127.0.0.1:6804/288278290,v1:127.0.0.1:6805/288278290] exists,up 846d515c-af3e-42ac-9df5-e0a3f3ad9bc8 2024-09-13T14:12:43.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:12:43.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:12:43.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:12:43.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-09-13T14:12:43.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:12:43.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:12:43.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:12:43.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T14:12:43.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:12:43.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:12:43.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:12:43.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:12:43.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:12:43.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:12:43.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:12:43.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:12:43.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:12:43.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:12:43.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:12:43.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:12:43.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:12:43.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:12:43.525 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:12:43.525 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:12:43.525 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:12:43.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:12:43.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:12:43.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:12:43.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:12:43.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:12:43.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:12:43.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:12:43.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:12:43.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:12:43.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:12:43.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:12:43.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:12:43.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:12:43.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:12:43.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:12:43.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d2a9e646-b47b-444c-bb41-10bb888f02f9 2024-09-13T14:12:43.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d2a9e646-b47b-444c-bb41-10bb888f02f9' 2024-09-13T14:12:43.529 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 d2a9e646-b47b-444c-bb41-10bb888f02f9 2024-09-13T14:12:43.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:12:43.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBbSORmrYAMIRAApfZRX+yzsetZn0nY8s0gBA== 2024-09-13T14:12:43.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBbSORmrYAMIRAApfZRX+yzsetZn0nY8s0gBA=="}' 2024-09-13T14:12:43.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d2a9e646-b47b-444c-bb41-10bb888f02f9 -i td/osd-scrub-repair/1/new.json 2024-09-13T14:12:44.087 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:12:44.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T14:12:44.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBbSORmrYAMIRAApfZRX+yzsetZn0nY8s0gBA== --osd-uuid d2a9e646-b47b-444c-bb41-10bb888f02f9 2024-09-13T14:12:44.150 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:44.151+0000 7f741381d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:12:44.163 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:44.167+0000 7f741381d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:12:44.164 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:44.167+0000 7f741381d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:12:44.165 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:44.167+0000 7f741381d7c0 -1 bdev(0x55b2604e1c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:12:44.165 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:44.167+0000 7f741381d7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T14:12:50.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T14:12:50.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:12:50.124 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:12:50.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T14:12:50.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:12:50.529 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:12:50.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T14:12:50.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:12:50.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:12:50.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:12:50.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:12:50.564 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:50.567+0000 7f1fdc68f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:12:50.592 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:50.595+0000 7f1fdc68f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:12:50.593 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:50.595+0000 7f1fdc68f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:12:50.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T14:12:50.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:12:50.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:12:50.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:12:50.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:12:50.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:50.877 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:12:50.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:12:50.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:12:50.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:51.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:51.673 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:51.675+0000 7f1fdc68f7c0 -1 Falling back to public interface 2024-09-13T14:12:52.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:52.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:52.218 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:12:52.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:12:52.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:52.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:12:52.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:53.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:53.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:53.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:12:53.555 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:12:53.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:53.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:12:53.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:54.358 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:12:54.359+0000 7f1fdc68f7c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:12:54.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:54.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:54.896 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:12:54.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:12:54.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:54.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:12:55.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:56.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:56.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:56.340 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:12:56.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:12:56.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:56.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:12:56.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:57.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:57.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:57.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:12:57.669 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:12:57.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:57.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:12:57.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:12:58.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:12:58.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:12:58.996 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:12:58.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:12:58.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:12:58.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:12:59.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:00.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:00.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:00.362 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:13:00.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:13:00.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:00.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:13:00.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:01.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:01.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:01.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:13:01.691 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:13:01.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:01.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:13:02.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:03.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:03.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:03.023 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:13:03.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:13:03.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:03.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:13:03.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:04.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:04.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:04.361 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:13:04.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:13:04.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:04.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:13:04.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:05.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:05.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:05.703 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:13:05.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:13:05.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:05.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:13:06.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:07.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:07.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:07.300 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:13:07.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:13:07.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:07.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:13:07.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:08.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:08.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:08.782 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:13:08.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:13:08.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:08.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:13:09.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:10.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:10.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:10.436 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:13:10.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:13:10.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:10.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:13:10.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:11.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:11.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:11.989 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:13:11.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:13:11.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:11.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:13:12.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:13.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:13.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:13.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:13:13.484 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:13:13.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:13.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:13:14.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:15.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:15.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:15.337 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:13:15.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:13:15.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:15.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:13:15.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:16.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:16.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:16.689 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:13:16.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:13:16.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:16.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:13:17.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:18.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:18.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:18.475 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:13:18.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:13:18.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:18.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:13:18.814 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1761196208,v1:127.0.0.1:6811/1761196208] [v2:127.0.0.1:6812/1761196208,v1:127.0.0.1:6813/1761196208] exists,up d2a9e646-b47b-444c-bb41-10bb888f02f9 2024-09-13T14:13:18.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:13:18.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:13:18.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:13:18.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-09-13T14:13:18.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:13:18.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:13:18.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:13:18.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T14:13:18.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:13:18.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:13:18.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:13:18.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:13:18.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:13:18.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:13:18.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T14:13:18.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T14:13:18.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:13:18.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:13:18.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:13:18.821 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:13:18.821 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:13:18.821 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:13:18.821 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:13:18.821 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:13:18.821 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:13:18.822 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:13:18.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:13:18.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:13:18.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:13:18.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:13:18.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:13:18.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:13:18.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:13:18.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:13:18.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:13:18.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:13:18.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:13:18.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:13:18.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T14:13:18.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:13:18.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2fe213ec-b5a8-47a5-9e43-e3c40019a90e 2024-09-13T14:13:18.823 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 2fe213ec-b5a8-47a5-9e43-e3c40019a90e 2024-09-13T14:13:18.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 2fe213ec-b5a8-47a5-9e43-e3c40019a90e' 2024-09-13T14:13:18.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:13:18.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB+SORmu2JJMhAAp8PLt85DdAQRGnWl+k4cMw== 2024-09-13T14:13:18.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB+SORmu2JJMhAAp8PLt85DdAQRGnWl+k4cMw=="}' 2024-09-13T14:13:18.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2fe213ec-b5a8-47a5-9e43-e3c40019a90e -i td/osd-scrub-repair/2/new.json 2024-09-13T14:13:19.198 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:13:19.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T14:13:19.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQB+SORmu2JJMhAAp8PLt85DdAQRGnWl+k4cMw== --osd-uuid 2fe213ec-b5a8-47a5-9e43-e3c40019a90e 2024-09-13T14:13:19.255 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:13:19.258+0000 7fe28de527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:13:19.274 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:13:19.274+0000 7fe28de527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:13:19.275 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:13:19.278+0000 7fe28de527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:13:19.276 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:13:19.278+0000 7fe28de527c0 -1 bdev(0x55782d1dbc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T14:13:19.276 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:13:19.278+0000 7fe28de527c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T14:13:25.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T14:13:25.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:13:25.178 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T14:13:25.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T14:13:25.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:13:25.615 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T14:13:25.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T14:13:25.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:13:25.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:13:25.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:13:25.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:13:25.652 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:13:25.654+0000 7f8e398a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:13:25.670 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:13:25.674+0000 7f8e398a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:13:25.672 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:13:25.674+0000 7f8e398a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:13:25.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T14:13:25.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:13:25.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:13:25.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:13:25.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:13:25.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:25.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:13:25.954 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:13:25.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:25.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:26.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:26.729 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:13:26.730+0000 7f8e398a57c0 -1 Falling back to public interface 2024-09-13T14:13:27.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:27.283 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:13:27.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:27.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:13:27.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:27.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:27.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:28.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:28.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:28.623 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:13:28.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:13:28.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:28.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:28.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:29.785 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:13:29.786+0000 7f8e398a57c0 -1 osd.2 0 log_to_monitors true 2024-09-13T14:13:29.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:29.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:29.962 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:13:29.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:13:29.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:29.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:30.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:31.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:31.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:31.350 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:13:31.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:13:31.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:31.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:31.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:32.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:32.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:32.690 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:13:32.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:13:32.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:32.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:33.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:34.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:34.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:34.025 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:13:34.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:13:34.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:34.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:34.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:35.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:35.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:35.370 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:13:35.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:13:35.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:35.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:35.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:36.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:36.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:36.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:13:36.713 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:13:36.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:36.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:37.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:38.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:38.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:38.047 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:13:38.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:13:38.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:38.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:38.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:39.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:39.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:39.381 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:13:39.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:13:39.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:39.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:39.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:40.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:40.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:40.724 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:13:40.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:13:40.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:40.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:41.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:42.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:42.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:42.055 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:13:42.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:13:42.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:42.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:42.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:43.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:43.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:43.744 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:13:43.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:13:43.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:43.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:44.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:45.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:45.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:45.159 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:13:45.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:13:45.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:45.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:45.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:46.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:46.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:46.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:13:46.546 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:13:46.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:46.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:47.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:48.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:48.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:48.309 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:13:48.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:13:48.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:48.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:48.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:49.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:49.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:49.969 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:13:49.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:13:49.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:49.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:50.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:13:51.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:13:51.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:13:51.701 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:13:51.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:13:51.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:13:51.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:13:52.097 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1428340111,v1:127.0.0.1:6819/1428340111] [v2:127.0.0.1:6820/1428340111,v1:127.0.0.1:6821/1428340111] exists,up 2fe213ec-b5a8-47a5-9e43-e3c40019a90e 2024-09-13T14:13:52.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:13:52.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:13:52.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:13:52.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-09-13T14:13:52.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-13T14:13:52.537 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'testpool' created 2024-09-13T14:13:52.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:13:53.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-09-13T14:13:54.027 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 1 size to 2 2024-09-13T14:13:54.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-09-13T14:13:54.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:13:54.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:13:54.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:13:54.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:13:54.047 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:13:54.048 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:13:54.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:13:54.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:13:54.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:13:54.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:13:54.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:13:54.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:13:54.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:13:54.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:13:54.156 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:13:54.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:13:54.575 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:13:54.575 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:13:54.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:13:54.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:13:54.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:13:54.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836498 2024-09-13T14:13:54.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836498 2024-09-13T14:13:54.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836498' 2024-09-13T14:13:54.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:13:54.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:13:54.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672973 2024-09-13T14:13:54.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672973 2024-09-13T14:13:54.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836498 1-42949672973' 2024-09-13T14:13:54.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:13:54.886 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:13:55.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509445 2024-09-13T14:13:55.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509445 2024-09-13T14:13:55.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836498 1-42949672973 2-64424509445' 2024-09-13T14:13:55.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:13:55.102 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836498 2024-09-13T14:13:55.103 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:13:55.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:13:55.104 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836498 2024-09-13T14:13:55.104 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:13:55.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836498 2024-09-13T14:13:55.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-13T14:13:55.106 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836498 2024-09-13T14:13:55.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:13:55.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836498 -lt 21474836498 2024-09-13T14:13:55.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:13:55.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672973 2024-09-13T14:13:55.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:13:55.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:13:55.509 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672973 2024-09-13T14:13:55.509 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:13:55.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672973 2024-09-13T14:13:55.510 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672973 2024-09-13T14:13:55.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-13T14:13:55.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:13:55.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-13T14:13:55.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:13:55.829 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509445 2024-09-13T14:13:55.829 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:13:55.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:13:55.831 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509445 2024-09-13T14:13:55.831 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:13:55.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509445 2024-09-13T14:13:55.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-13T14:13:55.833 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509445 2024-09-13T14:13:55.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:13:56.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509445 -lt 64424509445 2024-09-13T14:13:56.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:13:56.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:13:56.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:13:56.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:13:56.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:13:56.567 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:13:56.567 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:13:56.567 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:13:56.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:13:56.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:13:56.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:13:56.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T14:13:56.903 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:13:56.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:13:56.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:13:57.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T14:13:57.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T14:13:57.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T14:13:57.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T14:13:57.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:13:57.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:13:57.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:13:57.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:13:57.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:13:57.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:13:57.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:13:57.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:13:57.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:13:57.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:13:57.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T14:13:57.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:13:57.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:13:57.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:13:58.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T14:13:58.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T14:13:58.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T14:13:58.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T14:13:58.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T14:13:58.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T14:13:58.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T14:13:58.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T14:13:58.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T14:13:58.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T14:13:58.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T14:13:58.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T14:13:58.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:13:58.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T14:13:58.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:13:58.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:13:58.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:13:58.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:13:58.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:13:58.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:13:58.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:13:58.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:13:59.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T14:13:59.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:13:59.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:13:59.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:13:59.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T14:13:59.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T14:13:59.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T14:13:59.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T14:13:59.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T14:13:59.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T14:13:59.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T14:13:59.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T14:13:59.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T14:13:59.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T14:13:59.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T14:13:59.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-13T14:13:59.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:13:59.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-13T14:14:00.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:14:00.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:14:00.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:14:00.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:14:00.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:14:00.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:14:00.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:14:00.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:14:00.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:14:00.516 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:14:00.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:14:00.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:14:00.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:14:00.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:14:00.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:14:00.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-09-13T14:14:00.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-09-13T14:14:00.906 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-09-13T14:14:00.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-13T14:14:00.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:14:01.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-13T14:14:01.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:14:01.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-13T14:14:01.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:14:01.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-13T14:14:01.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:14:01.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-13T14:14:01.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:14:01.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-13T14:14:01.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:14:01.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-13T14:14:01.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:14:01.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-13T14:14:01.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:14:02.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-13T14:14:02.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:14:02.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-13T14:14:02.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:14:02.373 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-09-13T14:14:02.373 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-13T14:14:02.373 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:14:02.374 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-13T14:14:02.374 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:14:02.374 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:14:02.374 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:14:02.374 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:14:02.706 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T14:14:02.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:14:02.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T14:14:03.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-09-13T14:14:03.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:14:03.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:14:03.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T14:14:03.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:14:03.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-09-13T14:14:03.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:14:03.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:14:03.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T14:14:03.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:14:03.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T14:14:03.051 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:14:03.051 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:14:03.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:14:03.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:14:03.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:14:03.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:14:03.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-09-13T14:14:03.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:14:03.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:14:03.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T14:14:03.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:14:03.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:14:03.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-09-13T14:14:04.876 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-13T14:14:05.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:14:05.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:14:05.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:14:05.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T14:14:05.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:14:05.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:14:05.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:14:05.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:14:05.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:14:05.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:14:05.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:14:05.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:14:05.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:14:05.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:14:05.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:14:05.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:14:05.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:14:05.424 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:14:05.424 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:14:05.424 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:14:05.424 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:14:05.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:14:05.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:14:05.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:14:05.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:14:05.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:14:05.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:14:05.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:14:05.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:14:05.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:14:05.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:14:05.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:14:05.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:14:05.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T14:14:05.426 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T14:14:05.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:14:05.427 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T14:14:05.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T14:14:05.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:14:05.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:14:05.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:14:05.463 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:05.465+0000 7f178a95d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:14:05.472 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:05.473+0000 7f178a95d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:14:05.474 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:05.477+0000 7f178a95d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:14:05.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T14:14:05.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:14:05.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:14:05.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:14:05.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:14:05.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:05.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:14:05.760 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:14:05.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:05.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:14:06.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:14:06.509 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:06.509+0000 7f178a95d7c0 -1 Falling back to public interface 2024-09-13T14:14:07.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:14:07.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:07.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:14:07.107 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:14:07.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:07.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:14:07.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:14:08.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:14:08.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:08.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:14:08.443 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:14:08.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:08.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:14:08.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:14:09.209 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:09.213+0000 7f178a95d7c0 -1 osd.0 21 log_to_monitors true 2024-09-13T14:14:09.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:14:09.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:09.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:14:09.791 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:14:09.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:09.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:14:10.141 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/2919886316,v1:127.0.0.1:6803/2919886316] [v2:127.0.0.1:6804/2919886316,v1:127.0.0.1:6805/2919886316] exists,up 846d515c-af3e-42ac-9df5-e0a3f3ad9bc8 2024-09-13T14:14:10.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:14:10.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:14:10.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:14:10.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:14:10.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:14:10.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:14:10.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:14:10.142 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:14:10.142 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:14:10.142 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:14:10.142 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:14:10.142 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:14:10.143 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:14:10.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:14:10.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:14:10.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:14:10.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:14:10.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:14:10.249 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:14:10.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:14:10.583 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:14:10.583 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:14:10.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:14:10.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:10.584 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:14:10.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182402 2024-09-13T14:14:10.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182402 2024-09-13T14:14:10.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402' 2024-09-13T14:14:10.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:10.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:14:10.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-13T14:14:10.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-13T14:14:10.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672977' 2024-09-13T14:14:10.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:10.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:14:11.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509449 2024-09-13T14:14:11.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509449 2024-09-13T14:14:11.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672977 2-64424509449' 2024-09-13T14:14:11.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:11.081 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182402 2024-09-13T14:14:11.081 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:11.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:14:11.082 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182402 2024-09-13T14:14:11.082 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:11.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182402 2024-09-13T14:14:11.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-13T14:14:11.083 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 107374182402 2024-09-13T14:14:11.083 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:14:11.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-13T14:14:11.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:11.410 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-13T14:14:11.410 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:11.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:14:11.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-13T14:14:11.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:11.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-13T14:14:11.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-13T14:14:11.417 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672977 2024-09-13T14:14:11.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:14:11.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-13T14:14:11.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:11.809 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509449 2024-09-13T14:14:11.809 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:11.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:14:11.811 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:11.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509449 2024-09-13T14:14:11.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509449 2024-09-13T14:14:11.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-13T14:14:11.813 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509449 2024-09-13T14:14:11.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:14:12.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-13T14:14:12.141 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:14:12.142 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:14:12.142 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:14:12.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:14:12.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:14:12.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:14:12.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:14:12.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:14:12.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:14:12.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:14:12.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:14:12.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:14:12.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:14:12.864 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:14:12.864 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:14:13.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:14:13.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:14:13.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:14:13.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-09-13T14:14:13.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-13T14:14:13.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:14:13.234 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-13T14:14:13.234 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:14:13.234 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:14:13.235 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:14:13.235 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:14:13.583 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T14:14:13.583 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:14:13.583 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T14:14:13.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-09-13T14:14:13.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:14:13.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:14:13.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T14:14:13.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:14:13.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-09-13T14:14:13.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:14:13.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:14:13.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T14:14:13.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:14:13.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T14:14:13.924 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:14:13.924 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:14:13.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:14:13.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:14:13.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:14:14.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:14:14.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-13T14:14:14.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:14:14.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:14:14.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T14:14:14.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:14:14.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:14:14.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-13T14:14:15.658 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:104778fc:::obj2:head# 2024-09-13T14:14:16.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:14:16.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:14:16.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:14:16.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T14:14:16.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:14:16.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:14:16.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:14:16.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:14:16.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:14:16.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:14:16.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:14:16.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:14:16.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:14:16.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:14:16.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:14:16.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:14:16.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:14:16.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:14:16.203 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:14:16.203 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:14:16.203 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:14:16.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:14:16.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:14:16.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:14:16.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:14:16.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:14:16.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:14:16.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:14:16.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:14:16.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:14:16.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:14:16.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:14:16.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:14:16.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T14:14:16.206 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T14:14:16.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:14:16.206 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T14:14:16.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T14:14:16.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:14:16.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:14:16.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:14:16.242 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:16.245+0000 7ff234ae47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:14:16.245 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:16.245+0000 7ff234ae47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:14:16.246 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:16.249+0000 7ff234ae47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:14:16.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T14:14:16.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:14:16.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:14:16.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:14:16.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:14:16.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:16.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:14:16.547 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:14:16.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:14:16.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:16.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:14:17.497 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:17.497+0000 7ff234ae47c0 -1 Falling back to public interface 2024-09-13T14:14:17.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:14:17.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:17.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:14:17.886 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:14:17.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:17.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:14:18.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:14:19.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:14:19.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:19.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:14:19.213 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:14:19.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:19.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:14:19.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:14:20.383 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:20.385+0000 7ff234ae47c0 -1 osd.0 26 log_to_monitors true 2024-09-13T14:14:20.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:14:20.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:20.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:14:20.564 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:14:20.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:20.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:14:20.903 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/3914673152,v1:127.0.0.1:6803/3914673152] [v2:127.0.0.1:6804/3914673152,v1:127.0.0.1:6805/3914673152] exists,up 846d515c-af3e-42ac-9df5-e0a3f3ad9bc8 2024-09-13T14:14:20.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:14:20.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:14:20.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:14:20.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:14:20.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:14:20.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:14:20.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:14:20.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:14:20.904 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:14:20.904 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:14:20.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:14:20.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:14:20.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:14:21.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:14:21.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:14:21.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:14:21.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:14:21.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:14:21.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:14:21.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:14:21.361 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:14:21.362 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:14:21.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:14:21.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:21.363 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:14:21.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018882 2024-09-13T14:14:21.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018882 2024-09-13T14:14:21.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882' 2024-09-13T14:14:21.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:21.553 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:14:21.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672980 2024-09-13T14:14:21.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672980 2024-09-13T14:14:21.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672980' 2024-09-13T14:14:21.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:21.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:14:21.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-13T14:14:21.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-13T14:14:21.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672980 2-64424509452' 2024-09-13T14:14:21.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:21.920 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018882 2024-09-13T14:14:21.920 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:21.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:14:21.921 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018882 2024-09-13T14:14:21.921 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:21.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018882 2024-09-13T14:14:21.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-09-13T14:14:21.923 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 128849018882 2024-09-13T14:14:21.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:14:22.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 128849018882 2024-09-13T14:14:22.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:14:23.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:14:23.254 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:14:23.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-13T14:14:23.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:23.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672980 2024-09-13T14:14:23.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:23.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:14:23.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672980 2024-09-13T14:14:23.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:23.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672980 2024-09-13T14:14:23.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-13T14:14:23.597 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672980 2024-09-13T14:14:23.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:14:23.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-13T14:14:23.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:23.931 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-13T14:14:23.931 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:23.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:14:23.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-13T14:14:23.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:23.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-13T14:14:23.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-13T14:14:23.935 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509452 2024-09-13T14:14:23.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:14:24.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509453 -lt 64424509452 2024-09-13T14:14:24.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:14:24.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:14:24.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:14:24.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:14:24.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:14:24.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:14:24.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:14:24.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:14:24.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:14:24.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:14:24.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:14:24.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T14:14:24.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:14:24.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:14:24.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:14:25.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T14:14:25.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T14:14:25.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T14:14:25.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T14:14:25.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:14:25.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:14:25.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:14:25.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:14:25.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:14:25.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:14:25.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:14:25.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:14:25.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:14:25.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:14:25.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:14:25.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:14:25.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:14:25.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:14:26.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:14:26.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:14:26.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:14:26.137 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-09-13T14:14:26.137 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:14:26.137 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:14:26.137 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:14:26.138 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:14:26.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-13T14:14:26.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:14:26.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:14:26.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:14:26.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:14:26.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-13T14:14:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:14:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:14:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:14:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:14:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:14:26.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:14:26.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:14:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:14:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:14:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:14:26.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:14:26.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-13T14:14:26.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:14:26.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:14:26.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:14:26.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:14:26.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:14:26.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-09-13T14:14:28.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:14:28.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:14:28.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:14:28.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:14:28.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:14:28.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:14:28.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:14:28.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:14:28.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:14:28.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:14:28.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:14:28.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:14:28.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:14:28.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:14:28.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:14:28.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:14:28.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:14:28.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:14:28.934 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:14:28.934 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:14:28.934 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:14:28.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:14:28.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:14:28.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:14:28.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:14:28.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:14:28.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:14:28.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:14:28.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:14:28.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:14:28.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:14:28.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:14:28.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:14:28.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:14:28.936 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T14:14:28.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:14:28.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:14:28.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:14:28.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:14:28.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:14:28.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:14:28.972 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:28.972+0000 7f848ca077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:14:28.975 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:28.976+0000 7f848ca077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:14:28.976 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:28.976+0000 7f848ca077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:14:29.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:14:29.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:14:29.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:14:29.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:14:29.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:14:29.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:29.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:14:29.272 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:14:29.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:29.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:14:29.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:14:30.013 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:30.012+0000 7f848ca077c0 -1 Falling back to public interface 2024-09-13T14:14:30.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:14:30.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:30.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:14:30.591 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:14:30.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:30.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:14:30.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:14:31.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:14:31.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:31.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:14:31.920 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:14:31.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:31.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:14:32.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:14:32.747 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:32.748+0000 7f848ca077c0 -1 osd.1 31 log_to_monitors true 2024-09-13T14:14:33.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:14:33.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:33.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:14:33.266 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:14:33.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:33.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:14:33.723 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/4166228321,v1:127.0.0.1:6811/4166228321] [v2:127.0.0.1:6812/4166228321,v1:127.0.0.1:6813/4166228321] exists,up d2a9e646-b47b-444c-bb41-10bb888f02f9 2024-09-13T14:14:33.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:14:33.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:14:33.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:14:33.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:14:33.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:14:33.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:14:33.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:14:33.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:14:33.724 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:14:33.724 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:14:33.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:14:33.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:14:33.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:14:33.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:14:33.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:14:33.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:14:33.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:14:33.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:14:33.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:14:34.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:14:34.254 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:14:34.254 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:14:34.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:14:34.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:34.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:14:34.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018885 2024-09-13T14:14:34.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018885 2024-09-13T14:14:34.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018885' 2024-09-13T14:14:34.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:34.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:14:34.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-13T14:14:34.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-13T14:14:34.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018885 1-150323855362' 2024-09-13T14:14:34.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:34.557 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:14:34.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509456 2024-09-13T14:14:34.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509456 2024-09-13T14:14:34.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018885 1-150323855362 2-64424509456' 2024-09-13T14:14:34.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:34.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018885 2024-09-13T14:14:34.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:34.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:14:34.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018885 2024-09-13T14:14:34.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:34.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018885 2024-09-13T14:14:34.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-09-13T14:14:34.703 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 128849018885 2024-09-13T14:14:34.703 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:14:35.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018884 -lt 128849018885 2024-09-13T14:14:35.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:14:36.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:14:36.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:14:36.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018885 -lt 128849018885 2024-09-13T14:14:36.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:36.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-150323855362 2024-09-13T14:14:36.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:36.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:14:36.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-150323855362 2024-09-13T14:14:36.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:36.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-13T14:14:36.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-09-13T14:14:36.333 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 150323855362 2024-09-13T14:14:36.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:14:36.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-13T14:14:36.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:36.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509456 2024-09-13T14:14:36.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:36.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:14:36.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509456 2024-09-13T14:14:36.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:36.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509456 2024-09-13T14:14:36.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-13T14:14:36.656 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509456 2024-09-13T14:14:36.656 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:14:36.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-13T14:14:36.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:14:36.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:14:36.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:14:37.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:14:37.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:14:37.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:14:37.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:14:37.379 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:14:37.379 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:14:37.379 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:14:37.379 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:14:37.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:14:37.734 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:14:37.734 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:14:37.734 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:14:38.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:14:38.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:14:38.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:14:38.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-09-13T14:14:38.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-13T14:14:38.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2024-09-13T14:14:38.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2024-09-13T14:14:38.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:14:38.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-09-13T14:14:38.603 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-09-13T14:14:38.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:14:38.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T14:14:38.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-13T14:14:38.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:14:38.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-09-13T14:14:38.942 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-09-13T14:14:38.942 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:14:38.942 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:14:38.942 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:14:38.942 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:14:39.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-09-13T14:13:52.491627+0000 2024-09-13T14:14:39.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-09-13T14:14:39.388 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T14:14:39.388 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T14:14:39.388 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T14:14:39.388 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-08-30T14:12:59.394464+0000" 2024-09-13T14:14:39.388 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:14:39.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-09-13T14:13:52.491627+0000 2024-09-13T14:14:39.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T14:14:39.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:13:52.491627+0000 2024-09-13T14:14:39.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:14:39.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:14:39.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:14:39.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:14:39.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:14:39.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:14:39.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:14:39.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:14:39.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:13:52.491627+0000 '>' 2024-09-13T14:13:52.491627+0000 2024-09-13T14:14:39.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:14:40.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:14:40.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:14:40.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:14:40.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:14:40.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:14:40.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:14:40.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:14:41.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:13:52.491627+0000 '>' 2024-09-13T14:13:52.491627+0000 2024-09-13T14:14:41.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:14:42.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:14:42.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:14:42.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:14:42.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:14:42.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:14:42.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:14:42.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:14:42.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:13:52.491627+0000 '>' 2024-09-13T14:13:52.491627+0000 2024-09-13T14:14:42.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:14:43.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:14:43.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:14:43.371 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:14:43.371 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:14:43.371 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:14:43.372 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:14:43.372 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:14:43.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:14:40.862927+0000 '>' 2024-09-13T14:13:52.491627+0000 2024-09-13T14:14:43.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:14:43.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-09-13T14:14:43.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:14:43.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:14:43.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:14:43.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:14:43.688 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:14:43.688 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:14:43.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:14:43.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:14:43.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:14:43.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:14:43.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:14:43.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:14:43.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:14:43.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:14:43.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:14:44.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:14:44.105 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:14:44.105 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:14:44.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:14:44.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:44.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:14:44.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018888 2024-09-13T14:14:44.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018888 2024-09-13T14:14:44.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018888' 2024-09-13T14:14:44.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:44.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:14:44.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855365 2024-09-13T14:14:44.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855365 2024-09-13T14:14:44.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018888 1-150323855365' 2024-09-13T14:14:44.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:44.419 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:14:44.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-13T14:14:44.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-13T14:14:44.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018888 1-150323855365 2-64424509459' 2024-09-13T14:14:44.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:44.562 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018888 2024-09-13T14:14:44.562 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:44.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:14:44.563 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018888 2024-09-13T14:14:44.563 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:44.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018888 2024-09-13T14:14:44.565 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 128849018888 2024-09-13T14:14:44.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018888' 2024-09-13T14:14:44.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:14:44.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018887 -lt 128849018888 2024-09-13T14:14:44.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:14:45.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:14:45.883 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:14:46.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018888 -lt 128849018888 2024-09-13T14:14:46.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:46.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-150323855365 2024-09-13T14:14:46.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:46.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:14:46.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-150323855365 2024-09-13T14:14:46.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:46.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855365 2024-09-13T14:14:46.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-09-13T14:14:46.217 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 150323855365 2024-09-13T14:14:46.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:14:46.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855365 -lt 150323855365 2024-09-13T14:14:46.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:46.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-13T14:14:46.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:46.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:14:46.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-13T14:14:46.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:46.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-13T14:14:46.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-13T14:14:46.540 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509459 2024-09-13T14:14:46.540 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:14:46.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-13T14:14:46.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:14:46.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:14:46.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:14:47.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:14:47.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:14:47.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:14:47.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:14:47.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:14:47.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:14:47.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:14:47.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:14:47.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:14:47.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:14:47.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:14:47.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:14:47.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:14:47.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:14:47.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:14:47.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-09-13T14:14:47.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:14:47.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:14:48.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:14:48.294 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:14:48.294 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:14:48.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:14:48.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:48.295 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:14:48.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018890 2024-09-13T14:14:48.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018890 2024-09-13T14:14:48.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018890' 2024-09-13T14:14:48.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:48.458 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:14:48.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855367 2024-09-13T14:14:48.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855367 2024-09-13T14:14:48.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018890 1-150323855367' 2024-09-13T14:14:48.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:48.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:14:48.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509461 2024-09-13T14:14:48.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509461 2024-09-13T14:14:48.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018890 1-150323855367 2-64424509461' 2024-09-13T14:14:48.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:48.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018890 2024-09-13T14:14:48.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:48.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:14:48.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018890 2024-09-13T14:14:48.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:48.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018890 2024-09-13T14:14:48.724 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 128849018890 2024-09-13T14:14:48.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018890' 2024-09-13T14:14:48.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:14:49.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018889 -lt 128849018890 2024-09-13T14:14:49.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:14:50.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:14:50.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:14:50.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018890 -lt 128849018890 2024-09-13T14:14:50.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:50.342 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-150323855367 2024-09-13T14:14:50.342 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:50.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:14:50.344 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-150323855367 2024-09-13T14:14:50.344 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:50.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855367 2024-09-13T14:14:50.346 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 150323855367 2024-09-13T14:14:50.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 150323855367' 2024-09-13T14:14:50.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:14:50.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855367 -lt 150323855367 2024-09-13T14:14:50.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:50.666 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509461 2024-09-13T14:14:50.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:50.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:14:50.669 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509461 2024-09-13T14:14:50.669 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:50.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509461 2024-09-13T14:14:50.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-13T14:14:50.671 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509461 2024-09-13T14:14:50.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:14:51.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-13T14:14:51.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-13T14:14:51.012 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:39.876+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-13T14:14:51.012 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:39.876+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-13T14:14:51.012 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:39.876+0000 7f8470991640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-09-13T14:14:51.012 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:39.876+0000 7f8470991640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-09-13T14:14:51.013 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:39.876+0000 7f8470991640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-13T14:14:51.013 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-13T14:14:51.013 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-13T14:14:51.013 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish: 3 errors. 0 errors fixed 2024-09-13T14:14:51.013 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish 3 error(s) present with no repair possible 2024-09-13T14:14:51.013 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-13T14:14:51.013 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-09-13T14:14:51.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-09-13T14:14:51.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-13T14:14:51.313 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T14:14:51.313 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-09-13T14:14:40.862987+0000 21'10 36:110 [1,0] 1 [1,0] 1 21'10 2024-09-13T14:14:40.862927+0000 21'10 2024-09-13T14:14:40.862927+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:14:40.862927+0000 10 0 2024-09-13T14:14:51.313 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T14:14:51.313 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T14:14:51.313 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:14:51.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-13T14:14:51.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-09-13T14:14:51.616 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:14:51.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-09-13T14:14:51.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-13T14:14:51.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-13T14:14:51.633 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2024-09-13T14:14:51.633 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:14:51.633 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T14:14:51.633 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-13T14:14:51.634 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:14:51.940 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T14:14:51.940 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2024-09-13T14:14:51.940 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T14:14:52.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-09-13T14:14:52.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:14:52.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:14:52.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T14:14:52.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:14:52.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-09-13T14:14:52.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:14:52.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:14:52.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T14:14:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:14:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T14:14:52.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:14:52.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:14:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:14:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:14:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:14:52.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:14:52.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-09-13T14:14:52.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:14:52.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:14:52.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T14:14:52.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:14:52.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:14:52.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-09-13T14:14:53.061 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:14:53.061 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:14:53.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:14:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:14:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:14:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T14:14:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:14:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:14:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:14:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:14:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:14:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:14:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:14:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:14:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:14:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:14:53.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:14:53.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:14:53.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:14:53.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:14:53.352 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:14:53.352 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:14:53.352 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:14:53.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:14:53.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:14:53.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:14:53.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:14:53.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:14:53.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:14:53.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:14:53.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:14:53.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:14:53.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:14:53.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:14:53.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:14:53.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T14:14:53.354 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T14:14:53.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:14:53.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T14:14:53.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T14:14:53.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:14:53.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:14:53.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:14:53.391 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:53.396+0000 7f17efafc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:14:53.407 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:53.412+0000 7f17efafc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:14:53.408 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:53.412+0000 7f17efafc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:14:53.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T14:14:53.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:14:53.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:14:53.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:14:53.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:14:53.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:53.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:14:53.691 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:14:53.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:53.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:14:54.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:14:54.708 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:54.712+0000 7f17efafc7c0 -1 Falling back to public interface 2024-09-13T14:14:55.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:14:55.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:55.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:14:55.034 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:14:55.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:55.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:14:55.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:14:56.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:14:56.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:56.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:14:56.351 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:14:56.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:56.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:14:56.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:14:57.670 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:14:57.676+0000 7f17efafc7c0 -1 osd.0 36 log_to_monitors true 2024-09-13T14:14:57.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:14:57.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:14:57.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:14:57.680 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:14:57.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:14:57.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:14:58.086 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6802/1664852913,v1:127.0.0.1:6803/1664852913] [v2:127.0.0.1:6804/1664852913,v1:127.0.0.1:6805/1664852913] exists,up 846d515c-af3e-42ac-9df5-e0a3f3ad9bc8 2024-09-13T14:14:58.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:14:58.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:14:58.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:14:58.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:14:58.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:14:58.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:14:58.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:14:58.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:14:58.089 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:14:58.089 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:14:58.089 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:14:58.089 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:14:58.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:14:58.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:14:58.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:14:58.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:14:58.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:14:58.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:14:58.193 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:14:58.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:14:58.527 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:14:58.527 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:14:58.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:14:58.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:58.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:14:58.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691842 2024-09-13T14:14:58.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691842 2024-09-13T14:14:58.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691842' 2024-09-13T14:14:58.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:58.754 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:14:58.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855370 2024-09-13T14:14:58.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855370 2024-09-13T14:14:58.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691842 1-150323855370' 2024-09-13T14:14:58.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:14:58.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:14:59.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509464 2024-09-13T14:14:59.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509464 2024-09-13T14:14:59.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691842 1-150323855370 2-64424509464' 2024-09-13T14:14:59.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:59.128 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-171798691842 2024-09-13T14:14:59.128 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:59.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:14:59.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-171798691842 2024-09-13T14:14:59.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:59.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691842 2024-09-13T14:14:59.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-09-13T14:14:59.134 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 171798691842 2024-09-13T14:14:59.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:14:59.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691842 -lt 171798691842 2024-09-13T14:14:59.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:59.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-150323855370 2024-09-13T14:14:59.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:59.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:14:59.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-150323855370 2024-09-13T14:14:59.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:59.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855370 2024-09-13T14:14:59.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 150323855370' 2024-09-13T14:14:59.496 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 150323855370 2024-09-13T14:14:59.497 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:14:59.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855370 -lt 150323855370 2024-09-13T14:14:59.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:14:59.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509464 2024-09-13T14:14:59.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:14:59.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:14:59.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509464 2024-09-13T14:14:59.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:14:59.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509464 2024-09-13T14:14:59.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-09-13T14:14:59.827 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509464 2024-09-13T14:14:59.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:15:00.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509464 -lt 64424509464 2024-09-13T14:15:00.155 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:15:00.155 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:15:00.156 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:15:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:15:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:15:00.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:15:00.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:15:00.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:15:00.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:15:00.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:15:00.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:15:00.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T14:15:00.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:15:00.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:15:00.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:15:01.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T14:15:01.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T14:15:01.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T14:15:01.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T14:15:01.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:15:01.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:15:01.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:15:01.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:15:01.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:15:01.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:15:01.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:15:01.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:15:01.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:15:01.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:15:01.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T14:15:01.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:15:01.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:15:01.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:15:02.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T14:15:02.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T14:15:02.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T14:15:02.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T14:15:02.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T14:15:02.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T14:15:02.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T14:15:02.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T14:15:02.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T14:15:02.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T14:15:02.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T14:15:02.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T14:15:02.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:15:02.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T14:15:02.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:15:02.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:15:02.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:15:02.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:15:02.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:15:02.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:15:02.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:15:02.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:15:02.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T14:15:03.000 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:15:03.000 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:15:03.000 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:15:03.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T14:15:03.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T14:15:03.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T14:15:03.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T14:15:03.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T14:15:03.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T14:15:03.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T14:15:03.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T14:15:03.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T14:15:03.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T14:15:03.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T14:15:03.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-13T14:15:03.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:15:03.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-13T14:15:04.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:15:04.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:15:04.161 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:15:04.162 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:15:04.162 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:15:04.162 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:15:04.162 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:15:04.162 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:15:04.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:15:04.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:15:04.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:15:04.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:15:04.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:15:04.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:15:04.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:15:04.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-09-13T14:15:04.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:15:04.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-13T14:15:04.924 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:39.876+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-13T14:15:04.924 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:39.876+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-13T14:15:04.924 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:39.876+0000 7f8470991640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-09-13T14:15:04.925 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:39.876+0000 7f8470991640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-09-13T14:15:04.925 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:39.876+0000 7f8470991640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-13T14:15:04.925 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-13T14:15:04.925 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-13T14:15:04.925 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish: 3 errors. 0 errors fixed 2024-09-13T14:15:04.925 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish 3 error(s) present with no repair possible 2024-09-13T14:15:04.925 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-13T14:15:04.925 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-09-13T14:15:04.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-09-13T14:15:04.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:15:04.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:15:04.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:15:04.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:15:05.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-09-13T14:15:05.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:15:05.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:15:05.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:15:05.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:15:05.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-09-13T14:15:05.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:15:05.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:15:05.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:15:05.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:15:05.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:15:05.241 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:15:05.241 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:15:05.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:15:05.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:15:05.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:15:05.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:15:05.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-09-13T14:15:05.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:15:05.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:15:05.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:15:05.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:15:05.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:15:05.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-09-13T14:15:07.043 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-09-13T14:15:07.043 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:104778fc:::obj2:head# 2024-09-13T14:15:07.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:15:07.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:15:07.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:15:07.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:15:07.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:15:07.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:15:07.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:15:07.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:15:07.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:15:07.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:15:07.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:15:07.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:15:07.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:15:07.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:15:07.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:15:07.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:15:07.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:15:07.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:15:07.582 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:15:07.582 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:15:07.582 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:15:07.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:15:07.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:15:07.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:15:07.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:15:07.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:15:07.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:15:07.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:15:07.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:15:07.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:15:07.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:15:07.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:15:07.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:15:07.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:15:07.583 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T14:15:07.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:15:07.584 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:15:07.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:15:07.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:15:07.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:15:07.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:15:07.621 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:15:07.624+0000 7fb1b3b7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:15:07.623 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:15:07.628+0000 7fb1b3b7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:15:07.625 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:15:07.628+0000 7fb1b3b7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:15:07.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:15:07.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:15:07.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:15:07.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:15:07.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:15:07.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:07.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:15:07.910 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:15:07.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:07.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:15:08.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:09.084 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:15:09.088+0000 7fb1b3b7b7c0 -1 Falling back to public interface 2024-09-13T14:15:09.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:09.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:09.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:15:09.250 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:15:09.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:09.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:15:09.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:15:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:15:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:15:10.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:11.867 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:15:11.872+0000 7fb1b3b7b7c0 -1 osd.1 41 log_to_monitors true 2024-09-13T14:15:11.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:11.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:11.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:15:11.923 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:15:11.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:11.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:15:12.299 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 45 up_thru 40 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/3399792182,v1:127.0.0.1:6811/3399792182] [v2:127.0.0.1:6812/3399792182,v1:127.0.0.1:6813/3399792182] exists,up d2a9e646-b47b-444c-bb41-10bb888f02f9 2024-09-13T14:15:12.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:15:12.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:15:12.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:15:12.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:15:12.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:15:12.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:15:12.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:15:12.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:15:12.301 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:15:12.301 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:15:12.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:15:12.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:15:12.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:15:12.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:15:12.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:15:12.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:15:12.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:15:12.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:15:12.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:15:12.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:15:12.778 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:15:12.778 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:15:12.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:15:12.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:15:12.779 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:15:12.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691846 2024-09-13T14:15:12.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691846 2024-09-13T14:15:12.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691846' 2024-09-13T14:15:12.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:15:12.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:15:13.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528322 2024-09-13T14:15:13.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528322 2024-09-13T14:15:13.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691846 1-193273528322' 2024-09-13T14:15:13.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:15:13.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:15:13.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509468 2024-09-13T14:15:13.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509468 2024-09-13T14:15:13.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691846 1-193273528322 2-64424509468' 2024-09-13T14:15:13.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:15:13.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-171798691846 2024-09-13T14:15:13.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:15:13.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:15:13.258 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-171798691846 2024-09-13T14:15:13.258 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:15:13.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691846 2024-09-13T14:15:13.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 171798691846' 2024-09-13T14:15:13.259 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 171798691846 2024-09-13T14:15:13.260 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:15:13.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691846 -lt 171798691846 2024-09-13T14:15:13.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:15:13.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-193273528322 2024-09-13T14:15:13.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:15:13.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:15:13.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-193273528322 2024-09-13T14:15:13.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:15:13.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528322 2024-09-13T14:15:13.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2024-09-13T14:15:13.578 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 193273528322 2024-09-13T14:15:13.578 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:15:13.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528322 -lt 193273528322 2024-09-13T14:15:13.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:15:13.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509468 2024-09-13T14:15:13.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:15:13.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:15:13.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509468 2024-09-13T14:15:13.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:15:13.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509468 2024-09-13T14:15:13.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-09-13T14:15:13.906 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509468 2024-09-13T14:15:13.907 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:15:14.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509468 -lt 64424509468 2024-09-13T14:15:14.226 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:15:14.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:15:14.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:15:14.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:15:14.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:15:14.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:15:14.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:15:14.634 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:15:14.634 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:15:14.634 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:15:14.634 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:15:14.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:15:14.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:15:14.960 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:15:14.960 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:15:15.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:15:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:15:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:15:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-09-13T14:15:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-13T14:15:15.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-13T14:15:15.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:15:15.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:15:15.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:15:15.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:15:15.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:14:40.862927+0000 2024-09-13T14:15:15.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-13T14:15:16.002 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.0 on osd.1 to repair 2024-09-13T14:15:16.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-13T14:14:40.862927+0000 2024-09-13T14:15:16.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T14:15:16.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:14:40.862927+0000 2024-09-13T14:15:16.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:15:16.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:15:16.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:15:16.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:15:16.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:15:16.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:15:16.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:15:16.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:15:16.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:14:40.862927+0000 '>' 2024-09-13T14:14:40.862927+0000 2024-09-13T14:15:16.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:15:17.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:15:17.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:15:17.325 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:15:17.325 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:15:17.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:15:17.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:15:17.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:15:17.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:15:16.995916+0000 '>' 2024-09-13T14:14:40.862927+0000 2024-09-13T14:15:17.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:15:17.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:15:19.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-09-13T14:15:21.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-09-13T14:15:21.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:15:21.650 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:15:22.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:15:22.021 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:15:22.022 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:15:22.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:15:22.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:15:22.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:15:22.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691848 2024-09-13T14:15:22.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691848 2024-09-13T14:15:22.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691848' 2024-09-13T14:15:22.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:15:22.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:15:22.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528325 2024-09-13T14:15:22.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528325 2024-09-13T14:15:22.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691848 1-193273528325' 2024-09-13T14:15:22.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:15:22.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:15:22.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509471 2024-09-13T14:15:22.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509471 2024-09-13T14:15:22.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-171798691848 1-193273528325 2-64424509471' 2024-09-13T14:15:22.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:15:22.460 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-171798691848 2024-09-13T14:15:22.460 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:15:22.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:15:22.461 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-171798691848 2024-09-13T14:15:22.461 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:15:22.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691848 2024-09-13T14:15:22.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 171798691848' 2024-09-13T14:15:22.463 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 171798691848 2024-09-13T14:15:22.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:15:22.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691847 -lt 171798691848 2024-09-13T14:15:22.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:15:23.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:15:23.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:15:24.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691849 -lt 171798691848 2024-09-13T14:15:24.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:15:24.120 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-193273528325 2024-09-13T14:15:24.120 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:15:24.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:15:24.122 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-193273528325 2024-09-13T14:15:24.122 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:15:24.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528325 2024-09-13T14:15:24.124 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 193273528325 2024-09-13T14:15:24.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 193273528325' 2024-09-13T14:15:24.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:15:24.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528325 -lt 193273528325 2024-09-13T14:15:24.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:15:24.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509471 2024-09-13T14:15:24.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:15:24.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:15:24.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509471 2024-09-13T14:15:24.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:15:24.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509471 2024-09-13T14:15:24.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-09-13T14:15:24.449 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509471 2024-09-13T14:15:24.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:15:24.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509471 -lt 64424509471 2024-09-13T14:15:24.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-13T14:15:25.109 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T14:15:25.109 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-09-13T14:15:16.996060+0000 21'10 46:150 [1,0] 1 [1,0] 1 21'10 2024-09-13T14:15:16.995916+0000 21'10 2024-09-13T14:15:16.995916+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:15:16.995916+0000 9 0 2024-09-13T14:15:25.109 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T14:15:25.109 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T14:15:25.109 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:15:25.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-13T14:15:25.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-09-13T14:15:25.430 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:15:25.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-13T14:15:25.451 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:39.876+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-13T14:15:25.451 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:39.876+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-13T14:15:25.451 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:39.876+0000 7f8470991640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-09-13T14:15:25.451 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:39.876+0000 7f8470991640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-09-13T14:15:25.451 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:39.876+0000 7f8470991640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-13T14:15:25.451 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-13T14:15:25.451 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-13T14:15:25.451 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish: 3 errors. 0 errors fixed 2024-09-13T14:15:25.452 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish 3 error(s) present with no repair possible 2024-09-13T14:15:25.452 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-13T14:15:25.452 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:14:40.856+0000 7f8470991640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=16/16 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-09-13T14:15:25.452 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:15:16.991+0000 7fb197b05640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-09-13T14:15:25.452 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:15:16.991+0000 7fb197b05640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-13T14:15:25.452 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:15:16.991+0000 7fb197b05640 15 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-13T14:15:25.452 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:15:16.991+0000 7fb197b05640 20 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-09-13T14:15:25.453 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:15:16.991+0000 7fb197b05640 19 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=16/16 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-13T14:15:25.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T14:15:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:15:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:15:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:15:25.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:15:25.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:15:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:15:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:15:25.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:15:25.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:15:25.562 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:15:25.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:15:25.563 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:15:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:15:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:15:25.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:15:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:15:25.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:15:25.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:15:25.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:15:25.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:15:25.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:15:25.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:15:25.587 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:15:25.587 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:15:25.587 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:15:25.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:15:25.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:15:25.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:15:25.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T14:15:25.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T14:15:25.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T14:15:25.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T14:15:25.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:15:25.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:15:25.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:15:25.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:15:25.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:15:25.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:15:25.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:15:25.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:15:25.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:15:25.591 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:15:25.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:15:25.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:15:25.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:15:25.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:15:25.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:15:25.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:15:25.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:15:25.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:15:25.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:15:25.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:15:25.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:15:25.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:15:25.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:15:25.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:15:25.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:15:25.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:15:25.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:15:25.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:15:25.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T14:15:25.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:15:25.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:15:25.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:15:25.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T14:15:25.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:15:25.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T14:15:25.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:15:25.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T14:15:25.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-09-13T14:15:25.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-09-13T14:15:25.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-09-13T14:15:25.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-09-13T14:15:25.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T14:15:25.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T14:15:25.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T14:15:25.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T14:15:25.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T14:15:25.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T14:15:26.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T14:15:26.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:15:26.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:15:26.067 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:15:26.067 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:15:26.067 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:15:26.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:15:26.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T14:15:26.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T14:15:26.276 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T14:15:26.276 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:15:26.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:15:26.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T14:15:26.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T14:15:26.277 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:15:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:15:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:15:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:15:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:15:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:15:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:15:26.279 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:15:26.279 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T14:15:26.397 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T14:15:26.397 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:15:26.397 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:15:26.397 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T14:15:26.397 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T14:15:26.398 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:15:26.398 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:15:26.398 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:15:26.398 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:15:26.399 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:15:26.399 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:15:26.399 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:15:26.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:15:26.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T14:15:26.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-09-13T14:15:26.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T14:15:26.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T14:15:26.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T14:15:26.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T14:15:26.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T14:15:26.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T14:15:27.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T14:15:27.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:15:27.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:15:27.387 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:15:27.387 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:15:27.387 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:15:27.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:15:27.388 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:15:27.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:15:27.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:15:27.424 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-09-13T14:15:27.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-09-13T14:15:27.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:15:27.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:15:27.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:15:27.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T14:15:27.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:15:27.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:15:27.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:15:27.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:15:27.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:15:27.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:15:27.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:15:27.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:15:27.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:15:27.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:15:27.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:15:27.429 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:15:27.429 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:15:27.429 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:15:27.429 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:15:27.429 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:15:27.429 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:15:27.430 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:15:27.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:15:27.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:15:27.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:15:27.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:15:27.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:15:27.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:15:27.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:15:27.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:15:27.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:15:27.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:15:27.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:15:27.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:15:27.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:15:27.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:15:27.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e9046055-4549-4c87-8348-880296f678ef 2024-09-13T14:15:27.431 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 e9046055-4549-4c87-8348-880296f678ef 2024-09-13T14:15:27.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e9046055-4549-4c87-8348-880296f678ef' 2024-09-13T14:15:27.432 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:15:27.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD/SORm+WwSGxAAlIk0MI+W0WlWVJIdY44ztw== 2024-09-13T14:15:27.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD/SORm+WwSGxAAlIk0MI+W0WlWVJIdY44ztw=="}' 2024-09-13T14:15:27.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e9046055-4549-4c87-8348-880296f678ef -i td/osd-scrub-repair/0/new.json 2024-09-13T14:15:27.705 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:15:27.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T14:15:27.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD/SORm+WwSGxAAlIk0MI+W0WlWVJIdY44ztw== --osd-uuid e9046055-4549-4c87-8348-880296f678ef 2024-09-13T14:15:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:15:27.763+0000 7f8f8bdf17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:15:27.764 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:15:27.767+0000 7f8f8bdf17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:15:27.765 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:15:27.767+0000 7f8f8bdf17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:15:27.765 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:15:27.767+0000 7f8f8bdf17c0 -1 bdev(0x55fa1eb31c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:15:27.765 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:15:27.767+0000 7f8f8bdf17c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T14:15:34.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T14:15:34.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:15:34.128 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:15:34.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T14:15:34.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:15:34.529 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:15:34.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T14:15:34.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:15:34.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:15:34.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:15:34.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:15:34.564 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:15:34.567+0000 7f76c660d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:15:34.594 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:15:34.599+0000 7f76c660d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:15:34.596 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:15:34.599+0000 7f76c660d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:15:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T14:15:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:15:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:15:34.879 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:15:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:15:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:15:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:15:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:35.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:35.904 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:15:35.907+0000 7f76c660d7c0 -1 Falling back to public interface 2024-09-13T14:15:36.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:36.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:36.241 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:15:36.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:15:36.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:36.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:36.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:37.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:37.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:37.578 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:15:37.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:15:37.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:37.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:37.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:38.891 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:15:38.895+0000 7f76c660d7c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:15:38.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:38.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:38.912 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:15:38.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:15:38.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:38.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:39.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:40.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:40.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:40.338 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:15:40.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:15:40.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:40.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:40.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:41.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:41.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:41.725 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:15:41.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:15:41.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:41.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:42.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:43.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:43.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:43.045 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:15:43.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:15:43.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:43.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:43.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:44.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:44.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:44.374 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:15:44.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:15:44.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:44.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:44.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:45.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:45.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:45.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:15:45.714 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:15:45.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:45.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:46.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:47.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:47.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:47.047 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:15:47.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:15:47.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:47.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:47.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:48.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:48.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:48.388 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:15:48.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:15:48.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:48.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:48.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:49.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:49.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:49.863 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:15:49.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:15:49.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:49.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:50.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:51.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:51.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:51.204 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:15:51.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:15:51.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:51.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:51.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:52.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:52.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:52.622 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:15:52.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:15:52.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:52.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:53.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:54.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:54.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:54.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:15:54.455 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:15:54.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:54.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:55.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:56.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:56.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:56.199 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:15:56.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:15:56.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:56.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:56.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:57.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:57.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:57.621 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:15:57.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:15:57.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:57.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:58.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:15:59.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:15:59.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:15:59.153 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:15:59.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:15:59.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:15:59.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:15:59.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:00.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:00.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:00.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:16:00.782 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:16:00.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:00.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:16:01.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:02.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:02.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:02.235 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:16:02.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:16:02.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:02.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:16:02.583 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4098957631,v1:127.0.0.1:6803/4098957631] [v2:127.0.0.1:6804/4098957631,v1:127.0.0.1:6805/4098957631] exists,up e9046055-4549-4c87-8348-880296f678ef 2024-09-13T14:16:02.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:16:02.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:16:02.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:16:02.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-09-13T14:16:02.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:16:02.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:16:02.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:16:02.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T14:16:02.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:16:02.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:16:02.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:16:02.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:16:02.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:16:02.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:16:02.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:16:02.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:16:02.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:16:02.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:16:02.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:16:02.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:16:02.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:16:02.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:16:02.588 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:16:02.589 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:16:02.589 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:16:02.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:16:02.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:16:02.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:16:02.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:16:02.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:16:02.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:16:02.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:16:02.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:16:02.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:16:02.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:16:02.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:16:02.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:16:02.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:16:02.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:16:02.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:16:02.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d6c2c069-9780-44cf-ba88-9d9e6484c055 2024-09-13T14:16:02.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d6c2c069-9780-44cf-ba88-9d9e6484c055' 2024-09-13T14:16:02.591 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 d6c2c069-9780-44cf-ba88-9d9e6484c055 2024-09-13T14:16:02.591 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:16:02.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAiSeRm5rjAJBAA2qWZONbaTIcBCR0FkiesHA== 2024-09-13T14:16:02.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAiSeRm5rjAJBAA2qWZONbaTIcBCR0FkiesHA=="}' 2024-09-13T14:16:02.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d6c2c069-9780-44cf-ba88-9d9e6484c055 -i td/osd-scrub-repair/1/new.json 2024-09-13T14:16:02.972 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:16:02.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T14:16:02.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAiSeRm5rjAJBAA2qWZONbaTIcBCR0FkiesHA== --osd-uuid d6c2c069-9780-44cf-ba88-9d9e6484c055 2024-09-13T14:16:03.029 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:03.031+0000 7f31f71c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:16:03.057 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:03.059+0000 7f31f71c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:16:03.058 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:03.063+0000 7f31f71c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:16:03.059 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:03.063+0000 7f31f71c87c0 -1 bdev(0x55dcd9b73c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:16:03.059 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:03.063+0000 7f31f71c87c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T14:16:09.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T14:16:09.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:16:09.369 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:16:09.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T14:16:09.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:16:09.784 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:16:09.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T14:16:09.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:16:09.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:16:09.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:16:09.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:16:09.820 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:09.823+0000 7fea680637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:16:09.844 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:09.847+0000 7fea680637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:16:09.845 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:09.851+0000 7fea680637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:16:10.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T14:16:10.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:16:10.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:16:10.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:16:10.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:16:10.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:10.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:16:10.123 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:16:10.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:10.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:10.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:10.896 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:10.898+0000 7fea680637c0 -1 Falling back to public interface 2024-09-13T14:16:11.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:11.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:11.446 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:16:11.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:16:11.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:11.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:11.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:12.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:12.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:12.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:16:12.779 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:16:12.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:12.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:13.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:13.660 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:13.662+0000 7fea680637c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:16:14.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:14.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:14.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:16:14.093 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:16:14.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:14.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:14.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:15.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:15.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:15.486 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:16:15.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:16:15.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:15.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:15.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:16.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:16.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:16.819 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:16:16.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:16:16.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:16.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:17.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:18.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:18.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:18.158 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:16:18.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:16:18.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:18.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:18.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:19.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:19.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:19.636 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:16:19.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:16:19.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:19.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:19.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:20.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:20.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:20.977 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:16:20.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:16:20.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:20.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:21.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:22.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:22.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:22.362 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:16:22.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:16:22.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:22.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:22.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:23.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:23.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:23.765 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:16:23.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:16:23.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:23.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:24.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:25.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:25.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:25.098 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:16:25.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:16:25.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:25.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:25.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:26.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:26.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:26.582 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:16:26.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:16:26.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:26.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:26.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:27.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:27.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:27.914 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:16:27.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:16:27.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:27.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:28.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:29.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:29.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:29.327 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:16:29.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:16:29.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:29.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:30.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:31.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:31.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:31.041 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:16:31.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:16:31.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:31.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:31.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:32.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:32.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:32.811 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:16:32.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:16:32.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:32.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:33.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:34.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:34.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:34.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:16:34.713 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:16:34.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:34.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:35.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:36.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:36.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:36.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:16:36.158 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:16:36.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:36.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:36.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:37.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:37.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:37.600 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:16:37.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:16:37.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:37.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:38.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:39.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:39.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:39.073 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T14:16:39.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T14:16:39.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:39.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:16:39.405 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2116799706,v1:127.0.0.1:6811/2116799706] [v2:127.0.0.1:6812/2116799706,v1:127.0.0.1:6813/2116799706] exists,up d6c2c069-9780-44cf-ba88-9d9e6484c055 2024-09-13T14:16:39.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:16:39.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:16:39.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:16:39.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-09-13T14:16:39.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:16:39.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:16:39.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:16:39.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T14:16:39.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:16:39.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:16:39.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:16:39.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:16:39.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:16:39.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:16:39.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T14:16:39.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T14:16:39.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:16:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:16:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:16:39.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:16:39.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:16:39.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:16:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:16:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:16:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:16:39.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:16:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:16:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:16:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:16:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:16:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:16:39.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:16:39.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:16:39.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:16:39.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:16:39.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:16:39.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:16:39.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:16:39.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T14:16:39.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:16:39.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=da0f8261-cd09-4240-acc3-46f990ab8711 2024-09-13T14:16:39.413 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 da0f8261-cd09-4240-acc3-46f990ab8711 2024-09-13T14:16:39.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 da0f8261-cd09-4240-acc3-46f990ab8711' 2024-09-13T14:16:39.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:16:39.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBHSeRmRBL1GRAAxVet07G900mJcfcEqPXS/g== 2024-09-13T14:16:39.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBHSeRmRBL1GRAAxVet07G900mJcfcEqPXS/g=="}' 2024-09-13T14:16:39.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new da0f8261-cd09-4240-acc3-46f990ab8711 -i td/osd-scrub-repair/2/new.json 2024-09-13T14:16:39.879 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:16:39.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T14:16:39.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBHSeRmRBL1GRAAxVet07G900mJcfcEqPXS/g== --osd-uuid da0f8261-cd09-4240-acc3-46f990ab8711 2024-09-13T14:16:39.939 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:39.942+0000 7f5a37daf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:16:39.981 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:39.986+0000 7f5a37daf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:16:39.983 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:39.986+0000 7f5a37daf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:16:39.983 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:39.986+0000 7f5a37daf7c0 -1 bdev(0x55f222b5bc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T14:16:39.984 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:39.986+0000 7f5a37daf7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T14:16:46.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T14:16:46.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:16:46.416 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T14:16:46.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T14:16:46.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:16:46.880 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T14:16:46.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T14:16:46.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:16:46.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:16:46.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:16:46.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:16:46.917 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:46.922+0000 7f1e4e9747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:16:46.945 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:46.946+0000 7f1e4e9747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:16:46.945 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:46.950+0000 7f1e4e9747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:16:47.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T14:16:47.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:16:47.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:16:47.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:16:47.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:16:47.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:47.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:16:47.205 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:16:47.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:47.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:16:47.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:48.247 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:48.250+0000 7f1e4e9747c0 -1 Falling back to public interface 2024-09-13T14:16:48.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:48.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:48.580 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:16:48.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:16:48.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:48.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:16:48.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:49.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:49.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:49.917 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:16:49.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:16:49.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:49.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:16:50.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:51.063 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:16:51.066+0000 7f1e4e9747c0 -1 osd.2 0 log_to_monitors true 2024-09-13T14:16:51.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:51.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:51.244 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:16:51.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:16:51.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:51.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:16:51.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:52.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:52.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:52.602 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:16:52.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:16:52.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:52.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:16:52.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:53.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:53.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:53.941 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:16:53.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:16:53.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:53.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:16:54.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:55.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:55.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:55.264 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:16:55.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:16:55.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:55.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:16:55.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:56.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:56.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:56.584 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:16:56.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:16:56.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:56.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:16:56.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:57.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:57.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:57.973 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:16:57.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:16:57.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:57.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:16:58.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:16:59.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:16:59.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:16:59.311 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:16:59.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:16:59.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:16:59.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:16:59.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:00.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:00.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:00.711 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:17:00.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:17:00.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:00.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:17:01.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:02.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:02.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:02.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:17:02.162 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:17:02.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:02.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:17:02.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:03.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:03.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:03.500 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:17:03.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:17:03.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:03.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:17:04.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:05.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:05.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:05.354 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:17:05.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:17:05.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:05.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:17:06.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:07.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:07.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:07.250 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:17:07.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:17:07.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:07.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:17:07.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:08.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:08.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:08.632 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:17:08.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:17:08.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:08.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:17:09.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:10.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:10.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:10.422 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:17:10.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:17:10.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:10.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:17:11.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:12.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:12.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:12.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:17:12.224 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:17:12.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:12.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:17:12.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:13.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:13.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:13.764 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:17:13.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:17:13.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:13.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:17:14.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:15.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:15.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:15.504 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:17:15.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:17:15.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:15.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:17:15.832 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/122130853,v1:127.0.0.1:6819/122130853] [v2:127.0.0.1:6820/122130853,v1:127.0.0.1:6821/122130853] exists,up da0f8261-cd09-4240-acc3-46f990ab8711 2024-09-13T14:17:15.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:17:15.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:17:15.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:17:15.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-09-13T14:17:15.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-13T14:17:16.403 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'testpool' created 2024-09-13T14:17:16.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:17:17.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-09-13T14:17:18.011 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 1 size to 2 2024-09-13T14:17:18.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-09-13T14:17:18.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:17:18.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:17:18.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:17:18.033 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:17:18.034 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:17:18.034 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:17:18.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:17:18.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:17:18.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:17:18.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:17:18.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:17:18.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:17:18.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:17:18.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:17:18.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:17:18.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:17:18.551 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:17:18.551 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:17:18.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:17:18.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:17:18.552 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:17:18.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-13T14:17:18.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-13T14:17:18.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-13T14:17:18.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:17:18.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:17:18.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672971 2024-09-13T14:17:18.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672971 2024-09-13T14:17:18.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672971' 2024-09-13T14:17:18.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:17:18.860 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:17:19.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509445 2024-09-13T14:17:19.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509445 2024-09-13T14:17:19.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672971 2-64424509445' 2024-09-13T14:17:19.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:17:19.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-13T14:17:19.005 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:17:19.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:17:19.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-13T14:17:19.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:17:19.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-13T14:17:19.007 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836502 2024-09-13T14:17:19.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-13T14:17:19.008 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:17:19.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836502 2024-09-13T14:17:19.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:17:20.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:17:20.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:17:20.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-13T14:17:20.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:17:20.656 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672971 2024-09-13T14:17:20.656 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:17:20.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:17:20.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672971 2024-09-13T14:17:20.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:17:20.659 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672971 2024-09-13T14:17:20.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672971 2024-09-13T14:17:20.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-09-13T14:17:20.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:17:20.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672972 -lt 42949672971 2024-09-13T14:17:20.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:17:20.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509445 2024-09-13T14:17:20.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:17:20.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:17:20.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509445 2024-09-13T14:17:20.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:17:20.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509445 2024-09-13T14:17:20.978 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509445 2024-09-13T14:17:20.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-13T14:17:20.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:17:21.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509445 -lt 64424509445 2024-09-13T14:17:21.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:17:21.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:17:21.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:17:21.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:17:21.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:17:21.735 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:17:21.735 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:17:21.735 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:17:21.735 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:17:21.735 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:17:21.735 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:17:22.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:17:22.065 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:17:22.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:17:22.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:17:22.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:17:22.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:17:22.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:17:22.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-09-13T14:17:22.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-09-13T14:17:22.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-09-13T14:17:22.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-13T14:17:22.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:17:22.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-13T14:17:22.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:17:22.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-13T14:17:22.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:17:22.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-13T14:17:22.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:17:23.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-13T14:17:23.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:17:23.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-13T14:17:23.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:17:23.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-13T14:17:23.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:17:23.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-13T14:17:23.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:17:23.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-13T14:17:23.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:17:23.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-13T14:17:23.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-13T14:17:23.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-09-13T14:17:23.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-13T14:17:23.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:17:23.991 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-13T14:17:23.991 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:17:23.991 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:17:23.992 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:17:23.992 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:17:24.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T14:17:24.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:17:24.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T14:17:24.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-09-13T14:17:24.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:17:24.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:17:24.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T14:17:24.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:17:24.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-09-13T14:17:24.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:17:24.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:17:24.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T14:17:24.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:17:24.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T14:17:24.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:17:24.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:17:24.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:17:24.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:17:24.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:17:24.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:17:24.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-09-13T14:17:24.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:17:24.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:17:24.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T14:17:24.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:17:24.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:17:24.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-09-13T14:17:26.853 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-13T14:17:27.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:17:27.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:17:27.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:17:27.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T14:17:27.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:17:27.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:17:27.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:17:27.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:17:27.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:17:27.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:17:27.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:17:27.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:17:27.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:17:27.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:17:27.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:17:27.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:17:27.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:17:27.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:17:27.408 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:17:27.408 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:17:27.408 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:17:27.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:17:27.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:17:27.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:17:27.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:17:27.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:17:27.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:17:27.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:17:27.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:17:27.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:17:27.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:17:27.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:17:27.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:17:27.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T14:17:27.411 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T14:17:27.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:17:27.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T14:17:27.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T14:17:27.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:17:27.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:17:27.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:17:27.448 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:17:27.453+0000 7f69488297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:17:27.452 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:17:27.453+0000 7f69488297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:17:27.453 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:17:27.457+0000 7f69488297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:17:27.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T14:17:27.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:17:27.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:17:27.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:17:27.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:17:27.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:27.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:17:27.754 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:17:27.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:27.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:17:28.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:28.487 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:17:28.489+0000 7f69488297c0 -1 Falling back to public interface 2024-09-13T14:17:29.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:29.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:29.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:17:29.108 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:17:29.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:29.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:17:29.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:30.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:30.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:30.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:17:30.439 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:17:30.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:30.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:17:30.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:31.393 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:17:31.397+0000 7f69488297c0 -1 osd.0 21 log_to_monitors true 2024-09-13T14:17:31.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:31.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:31.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:17:31.803 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:17:31.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:31.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:17:32.224 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/315175952,v1:127.0.0.1:6803/315175952] [v2:127.0.0.1:6804/315175952,v1:127.0.0.1:6805/315175952] exists,up e9046055-4549-4c87-8348-880296f678ef 2024-09-13T14:17:32.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:17:32.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:17:32.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:17:32.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:17:32.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:17:32.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:17:32.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:17:32.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:17:32.225 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:17:32.225 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:17:32.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:17:32.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:17:32.226 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:17:32.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:17:32.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:17:32.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:17:32.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:17:32.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:17:32.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:17:32.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:17:32.765 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:17:32.765 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:17:32.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:17:32.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:17:32.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:17:33.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182402 2024-09-13T14:17:33.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182402 2024-09-13T14:17:33.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402' 2024-09-13T14:17:33.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:17:33.033 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:17:33.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672975 2024-09-13T14:17:33.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672975 2024-09-13T14:17:33.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672975' 2024-09-13T14:17:33.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:17:33.182 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:17:33.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509449 2024-09-13T14:17:33.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509449 2024-09-13T14:17:33.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672975 2-64424509449' 2024-09-13T14:17:33.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:17:33.314 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182402 2024-09-13T14:17:33.314 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:17:33.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:17:33.315 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182402 2024-09-13T14:17:33.315 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:17:33.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182402 2024-09-13T14:17:33.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-13T14:17:33.317 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 107374182402 2024-09-13T14:17:33.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:17:33.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 107374182402 2024-09-13T14:17:33.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:17:34.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:17:34.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:17:34.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-13T14:17:34.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:17:34.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:17:34.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672975 2024-09-13T14:17:34.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:17:34.931 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672975 2024-09-13T14:17:34.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:17:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672975 2024-09-13T14:17:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-13T14:17:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672975 2024-09-13T14:17:34.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:17:35.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-13T14:17:35.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:17:35.275 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509449 2024-09-13T14:17:35.275 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:17:35.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:17:35.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509449 2024-09-13T14:17:35.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:17:35.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509449 2024-09-13T14:17:35.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-13T14:17:35.278 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509449 2024-09-13T14:17:35.278 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:17:35.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-13T14:17:35.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:17:35.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:17:35.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:17:36.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:17:36.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:17:36.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:17:36.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:17:36.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:17:36.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:17:36.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:17:36.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:17:36.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:17:36.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:17:36.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:17:36.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:17:36.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:17:36.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:17:36.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:17:36.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-09-13T14:17:36.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:17:36.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:17:36.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:17:36.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:17:37.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-09-13T14:17:37.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:17:37.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:17:37.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:17:37.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:17:37.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-09-13T14:17:37.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:17:37.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:17:37.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:17:37.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:17:37.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:17:37.086 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:17:37.086 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:17:37.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:17:37.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:17:37.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:17:37.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:17:37.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-09-13T14:17:37.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:17:37.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:17:37.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:17:37.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:17:37.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:17:37.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-09-13T14:17:39.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:17:39.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:17:39.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:17:39.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:17:39.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:17:39.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:17:39.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:17:39.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:17:39.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:17:39.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:17:39.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:17:39.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:17:39.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:17:39.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:17:39.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:17:39.665 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:17:39.665 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:17:39.665 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:17:39.666 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:17:39.666 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:17:39.666 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:17:39.666 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:17:39.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:17:39.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:17:39.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:17:39.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:17:39.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:17:39.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:17:39.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:17:39.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:17:39.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:17:39.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:17:39.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:17:39.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:17:39.668 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T14:17:39.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:17:39.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:17:39.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:17:39.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:17:39.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:17:39.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:17:39.704 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:17:39.705+0000 7fa9eb3fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:17:39.706 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:17:39.709+0000 7fa9eb3fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:17:39.708 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:17:39.709+0000 7fa9eb3fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:17:40.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:17:40.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:17:40.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:17:40.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:17:40.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:17:40.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:40.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:17:40.000 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:17:40.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:40.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:17:40.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:40.991 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:17:40.993+0000 7fa9eb3fb7c0 -1 Falling back to public interface 2024-09-13T14:17:41.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:41.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:41.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:17:41.365 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:17:41.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:41.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:17:41.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:42.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:42.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:42.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:17:42.703 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:17:42.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:42.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:17:43.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:43.900 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:17:43.901+0000 7fa9eb3fb7c0 -1 osd.1 26 log_to_monitors true 2024-09-13T14:17:44.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:44.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:44.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:17:44.043 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:17:44.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:44.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:17:44.507 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 30 up_thru 25 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/1672301742,v1:127.0.0.1:6811/1672301742] [v2:127.0.0.1:6812/1672301742,v1:127.0.0.1:6813/1672301742] exists,up d6c2c069-9780-44cf-ba88-9d9e6484c055 2024-09-13T14:17:44.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:17:44.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:17:44.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:17:44.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:17:44.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:17:44.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:17:44.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:17:44.508 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:17:44.508 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:17:44.508 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:17:44.509 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:17:44.509 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:17:44.509 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:17:44.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:17:44.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:17:44.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:17:44.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:17:44.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:17:44.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:17:45.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:17:45.008 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:17:45.008 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:17:45.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:17:45.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:17:45.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:17:45.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182405 2024-09-13T14:17:45.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182405 2024-09-13T14:17:45.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182405' 2024-09-13T14:17:45.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:17:45.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:17:45.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018882 2024-09-13T14:17:45.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018882 2024-09-13T14:17:45.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182405 1-128849018882' 2024-09-13T14:17:45.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:17:45.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:17:45.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-13T14:17:45.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-13T14:17:45.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182405 1-128849018882 2-64424509452' 2024-09-13T14:17:45.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:17:45.572 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182405 2024-09-13T14:17:45.572 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:17:45.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:17:45.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182405 2024-09-13T14:17:45.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:17:45.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182405 2024-09-13T14:17:45.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-09-13T14:17:45.574 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 107374182405 2024-09-13T14:17:45.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:17:45.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182404 -lt 107374182405 2024-09-13T14:17:45.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:17:46.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:17:46.889 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:17:47.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182405 -lt 107374182405 2024-09-13T14:17:47.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:17:47.210 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-128849018882 2024-09-13T14:17:47.210 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:17:47.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:17:47.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-128849018882 2024-09-13T14:17:47.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:17:47.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018882 2024-09-13T14:17:47.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-09-13T14:17:47.214 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 128849018882 2024-09-13T14:17:47.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:17:47.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-13T14:17:47.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:17:47.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-13T14:17:47.546 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:17:47.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:17:47.549 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-13T14:17:47.549 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:17:47.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-13T14:17:47.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-13T14:17:47.550 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509452 2024-09-13T14:17:47.550 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:17:47.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509453 -lt 64424509452 2024-09-13T14:17:47.903 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:17:47.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:17:47.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:17:48.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:17:48.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:17:48.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:17:48.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:17:48.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:17:48.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:17:48.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:17:48.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:17:48.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:17:48.613 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:17:48.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:17:48.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:17:48.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:17:48.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:17:48.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:17:48.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-09-13T14:17:48.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-13T14:17:48.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:17:48.991 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-13T14:17:48.992 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:17:48.992 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:17:48.992 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:17:48.992 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:17:49.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T14:17:49.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:17:49.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T14:17:49.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-09-13T14:17:49.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:17:49.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:17:49.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T14:17:49.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:17:49.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-09-13T14:17:49.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:17:49.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:17:49.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T14:17:49.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:17:49.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T14:17:49.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:17:49.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:17:49.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:17:49.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:17:49.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:17:49.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:17:49.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-13T14:17:49.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:17:49.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:17:49.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T14:17:49.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:17:49.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:17:49.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-13T14:17:51.544 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:104778fc:::obj2:head# 2024-09-13T14:17:52.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:17:52.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:17:52.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:17:52.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T14:17:52.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:17:52.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:17:52.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:17:52.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:17:52.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:17:52.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:17:52.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:17:52.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:17:52.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:17:52.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:17:52.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:17:52.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:17:52.086 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:17:52.086 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:17:52.086 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:17:52.086 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:17:52.086 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:17:52.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:17:52.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:17:52.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:17:52.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:17:52.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:17:52.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:17:52.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:17:52.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:17:52.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:17:52.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:17:52.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:17:52.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:17:52.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T14:17:52.088 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T14:17:52.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:17:52.088 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T14:17:52.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T14:17:52.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:17:52.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:17:52.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:17:52.126 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:17:52.129+0000 7f77e9d727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:17:52.129 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:17:52.133+0000 7f77e9d727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:17:52.130 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:17:52.133+0000 7f77e9d727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:17:52.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T14:17:52.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:17:52.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:17:52.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:17:52.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:17:52.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:52.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:17:52.421 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:17:52.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:52.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:17:52.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:53.423 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:17:53.425+0000 7f77e9d727c0 -1 Falling back to public interface 2024-09-13T14:17:53.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:53.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:53.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:17:53.765 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:17:53.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:53.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:17:54.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:55.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:55.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:55.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:17:55.095 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:17:55.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:55.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:17:55.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:56.247 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:17:56.249+0000 7f77e9d727c0 -1 osd.0 31 log_to_monitors true 2024-09-13T14:17:56.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:56.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:56.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:17:56.430 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:17:56.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:56.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:17:56.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:17:57.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:17:57.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:17:57.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:17:57.796 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:17:57.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:17:57.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:17:58.132 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 35 up_thru 27 down_at 32 last_clean_interval [25,31) [v2:127.0.0.1:6802/1525171643,v1:127.0.0.1:6803/1525171643] [v2:127.0.0.1:6804/1525171643,v1:127.0.0.1:6805/1525171643] exists,up e9046055-4549-4c87-8348-880296f678ef 2024-09-13T14:17:58.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:17:58.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:17:58.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:17:58.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:17:58.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:17:58.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:17:58.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:17:58.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:17:58.134 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:17:58.134 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:17:58.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:17:58.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:17:58.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:17:58.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:17:58.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:17:58.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:17:58.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:17:58.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:17:58.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:17:58.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:17:58.573 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:17:58.573 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:17:58.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:17:58.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:17:58.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:17:58.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-13T14:17:58.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-13T14:17:58.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362' 2024-09-13T14:17:58.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:17:58.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:17:58.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018885 2024-09-13T14:17:58.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018885 2024-09-13T14:17:58.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362 1-128849018885' 2024-09-13T14:17:58.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:17:58.860 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:17:59.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509456 2024-09-13T14:17:59.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509456 2024-09-13T14:17:59.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362 1-128849018885 2-64424509456' 2024-09-13T14:17:59.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:17:59.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855362 2024-09-13T14:17:59.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:17:59.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:17:59.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855362 2024-09-13T14:17:59.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:17:59.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-13T14:17:59.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-09-13T14:17:59.004 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 150323855362 2024-09-13T14:17:59.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:17:59.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 150323855362 2024-09-13T14:17:59.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:18:00.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:18:00.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:18:00.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-13T14:18:00.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:18:00.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-128849018885 2024-09-13T14:18:00.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:18:00.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:18:00.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-128849018885 2024-09-13T14:18:00.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:18:00.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018885 2024-09-13T14:18:00.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 128849018885' 2024-09-13T14:18:00.662 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 128849018885 2024-09-13T14:18:00.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:18:00.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018886 -lt 128849018885 2024-09-13T14:18:00.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:18:00.998 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509456 2024-09-13T14:18:00.998 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:18:00.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:18:01.000 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509456 2024-09-13T14:18:01.000 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:18:01.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509456 2024-09-13T14:18:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-13T14:18:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509456 2024-09-13T14:18:01.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:18:01.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-13T14:18:01.342 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:18:01.343 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:18:01.343 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:18:01.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:18:01.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:18:01.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:18:01.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:18:01.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:18:01.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:18:01.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:18:01.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:18:02.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:18:02.110 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:18:02.110 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:18:02.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:18:02.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:18:02.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:18:02.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:18:02.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-09-13T14:18:02.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:18:02.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:18:02.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:18:02.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:18:02.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-13T14:18:02.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:18:02.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:18:02.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:18:02.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:18:02.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-13T14:18:02.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:18:02.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:18:02.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:18:02.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:18:02.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:18:02.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:18:02.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:18:02.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:18:02.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:18:02.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:18:04.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:18:04.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-13T14:18:04.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:18:04.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:18:04.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:18:04.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:18:04.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:18:04.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-09-13T14:18:05.556 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-09-13T14:18:06.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:18:06.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:18:06.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:18:06.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:18:06.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:18:06.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:18:06.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:18:06.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:18:06.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:18:06.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:18:06.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:18:06.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:18:06.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:18:06.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:18:06.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:18:06.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:18:06.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:18:06.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:18:06.093 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:18:06.094 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:18:06.094 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:18:06.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:18:06.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:18:06.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:18:06.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:18:06.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:18:06.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:18:06.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:18:06.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:18:06.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:18:06.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:18:06.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:18:06.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:18:06.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:18:06.095 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T14:18:06.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:18:06.096 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:18:06.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:18:06.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:18:06.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:18:06.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:18:06.138 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:18:06.141+0000 7fa8d462f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:18:06.141 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:18:06.145+0000 7fa8d462f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:18:06.142 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:18:06.145+0000 7fa8d462f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:18:06.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:18:06.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:18:06.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:18:06.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:18:06.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:18:06.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:06.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:18:06.432 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:18:06.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:06.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:18:06.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:07.171 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:18:07.173+0000 7fa8d462f7c0 -1 Falling back to public interface 2024-09-13T14:18:07.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:07.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:07.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:18:07.780 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:18:07.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:07.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:18:08.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:09.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:09.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:09.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:18:09.120 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:18:09.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:09.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:18:09.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:09.838 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:18:09.841+0000 7fa8d462f7c0 -1 osd.1 36 log_to_monitors true 2024-09-13T14:18:10.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:10.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:10.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:18:10.445 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:18:10.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:10.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:18:10.875 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6810/1093294031,v1:127.0.0.1:6811/1093294031] [v2:127.0.0.1:6812/1093294031,v1:127.0.0.1:6813/1093294031] exists,up d6c2c069-9780-44cf-ba88-9d9e6484c055 2024-09-13T14:18:10.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:18:10.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:18:10.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:18:10.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:18:10.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:18:10.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:18:10.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:18:10.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:18:10.877 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:18:10.877 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:18:10.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:18:10.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:18:10.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:18:10.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:18:10.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:18:10.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:18:10.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:18:10.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:18:10.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:18:11.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:18:11.345 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:18:11.345 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:18:11.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:18:11.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:18:11.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:18:11.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855365 2024-09-13T14:18:11.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855365 2024-09-13T14:18:11.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855365' 2024-09-13T14:18:11.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:18:11.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:18:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691842 2024-09-13T14:18:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691842 2024-09-13T14:18:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855365 1-171798691842' 2024-09-13T14:18:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:18:11.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:18:11.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509460 2024-09-13T14:18:11.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509460 2024-09-13T14:18:11.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855365 1-171798691842 2-64424509460' 2024-09-13T14:18:11.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:18:11.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855365 2024-09-13T14:18:11.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:18:11.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:18:11.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855365 2024-09-13T14:18:11.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:18:11.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855365 2024-09-13T14:18:11.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855365' 2024-09-13T14:18:11.808 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 150323855365 2024-09-13T14:18:11.809 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:18:12.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855364 -lt 150323855365 2024-09-13T14:18:12.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:18:13.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:18:13.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:18:13.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855366 -lt 150323855365 2024-09-13T14:18:13.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:18:13.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-171798691842 2024-09-13T14:18:13.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:18:13.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:18:13.449 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-171798691842 2024-09-13T14:18:13.449 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:18:13.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691842 2024-09-13T14:18:13.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-09-13T14:18:13.450 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 171798691842 2024-09-13T14:18:13.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:18:13.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691842 -lt 171798691842 2024-09-13T14:18:13.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:18:13.782 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509460 2024-09-13T14:18:13.782 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:18:13.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:18:13.785 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509460 2024-09-13T14:18:13.785 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:18:13.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509460 2024-09-13T14:18:13.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-13T14:18:13.786 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509460 2024-09-13T14:18:13.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:18:14.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509460 -lt 64424509460 2024-09-13T14:18:14.143 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:18:14.143 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:18:14.143 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:18:14.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:18:14.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:18:14.557 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:18:14.557 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:18:14.557 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:18:14.557 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:18:14.558 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:18:14.558 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:18:14.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:18:14.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:18:14.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:18:14.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:18:15.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:18:15.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:18:15.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:18:15.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-09-13T14:18:15.445 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T14:18:15.445 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-13T14:18:15.445 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:18:15.457 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T14:18:15.457 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-13T14:18:15.457 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:18:15.469 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2: { 2024-09-13T14:18:15.469 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-09-13T14:18:15.470 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:18:15.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-09-13T14:18:15.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-13T14:18:15.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2024-09-13T14:18:15.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2024-09-13T14:18:15.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:18:15.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-09-13T14:18:15.783 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-09-13T14:18:15.783 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:18:15.783 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T14:18:15.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-13T14:18:15.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:18:16.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-09-13T14:18:16.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-09-13T14:18:16.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:18:16.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:18:16.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:18:16.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:18:16.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-09-13T14:17:16.293535+0000 2024-09-13T14:18:16.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-09-13T14:18:16.626 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T14:18:16.626 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T14:18:16.626 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T14:18:16.626 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-08-30T14:16:36.634263+0000" 2024-09-13T14:18:16.626 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:18:16.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-09-13T14:17:16.293535+0000 2024-09-13T14:18:16.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T14:18:16.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:17:16.293535+0000 2024-09-13T14:18:16.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:18:16.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:18:16.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:18:16.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:18:16.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:18:16.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:18:16.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:18:16.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:18:16.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:17:16.293535+0000 '>' 2024-09-13T14:17:16.293535+0000 2024-09-13T14:18:16.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:18:17.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:18:17.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:18:17.944 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:18:17.944 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:18:17.944 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:18:17.944 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:18:17.944 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:18:18.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:17:16.293535+0000 '>' 2024-09-13T14:17:16.293535+0000 2024-09-13T14:18:18.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:18:19.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:18:19.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:18:19.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:18:19.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:18:19.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:18:19.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:18:19.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:18:19.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:17:16.293535+0000 '>' 2024-09-13T14:17:16.293535+0000 2024-09-13T14:18:19.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:18:20.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:18:20.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:18:20.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:18:20.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:18:20.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:18:20.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:18:20.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:18:20.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:18:16.919425+0000 '>' 2024-09-13T14:17:16.293535+0000 2024-09-13T14:18:20.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:18:20.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-09-13T14:18:20.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:18:20.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:18:20.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:18:20.900 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:18:20.901 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:18:20.901 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:18:20.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:18:20.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:18:20.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:18:21.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:18:21.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:18:21.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:18:21.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:18:21.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:18:21.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:18:21.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:18:21.325 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:18:21.325 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:18:21.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:18:21.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:18:21.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:18:21.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855368 2024-09-13T14:18:21.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855368 2024-09-13T14:18:21.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855368' 2024-09-13T14:18:21.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:18:21.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:18:21.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691845 2024-09-13T14:18:21.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691845 2024-09-13T14:18:21.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855368 1-171798691845' 2024-09-13T14:18:21.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:18:21.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:18:21.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509463 2024-09-13T14:18:21.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509463 2024-09-13T14:18:21.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855368 1-171798691845 2-64424509463' 2024-09-13T14:18:21.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:18:21.767 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855368 2024-09-13T14:18:21.768 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:18:21.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:18:21.769 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855368 2024-09-13T14:18:21.769 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:18:21.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855368 2024-09-13T14:18:21.770 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 150323855368 2024-09-13T14:18:21.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855368' 2024-09-13T14:18:21.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:18:22.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855367 -lt 150323855368 2024-09-13T14:18:22.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:18:23.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:18:23.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:18:23.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855369 -lt 150323855368 2024-09-13T14:18:23.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:18:23.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-171798691845 2024-09-13T14:18:23.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:18:23.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:18:23.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-171798691845 2024-09-13T14:18:23.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:18:23.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691845 2024-09-13T14:18:23.405 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 171798691845 2024-09-13T14:18:23.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-09-13T14:18:23.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:18:23.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691845 -lt 171798691845 2024-09-13T14:18:23.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:18:23.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509463 2024-09-13T14:18:23.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:18:23.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:18:23.745 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509463 2024-09-13T14:18:23.745 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:18:23.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509463 2024-09-13T14:18:23.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-09-13T14:18:23.747 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509463 2024-09-13T14:18:23.747 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:18:24.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509463 -lt 64424509463 2024-09-13T14:18:24.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:18:24.080 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:18:24.080 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:18:24.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:18:24.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:18:24.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:18:24.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:18:24.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:18:24.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:18:24.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:18:24.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:18:24.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:18:24.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:18:24.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:18:24.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:18:25.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:18:25.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:18:25.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:18:25.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-09-13T14:18:25.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:18:25.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:18:25.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:18:25.556 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:18:25.556 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:18:25.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:18:25.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:18:25.556 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:18:25.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855370 2024-09-13T14:18:25.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855370 2024-09-13T14:18:25.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855370' 2024-09-13T14:18:25.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:18:25.690 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:18:25.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691847 2024-09-13T14:18:25.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691847 2024-09-13T14:18:25.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855370 1-171798691847' 2024-09-13T14:18:25.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:18:25.820 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:18:25.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509465 2024-09-13T14:18:25.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509465 2024-09-13T14:18:25.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855370 1-171798691847 2-64424509465' 2024-09-13T14:18:25.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:18:25.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855370 2024-09-13T14:18:25.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:18:25.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:18:25.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855370 2024-09-13T14:18:25.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:18:25.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855370 2024-09-13T14:18:25.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855370' 2024-09-13T14:18:25.959 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 150323855370 2024-09-13T14:18:25.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:18:26.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855370 -lt 150323855370 2024-09-13T14:18:26.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:18:26.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-171798691847 2024-09-13T14:18:26.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:18:26.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:18:26.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-171798691847 2024-09-13T14:18:26.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:18:26.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691847 2024-09-13T14:18:26.263 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 171798691847 2024-09-13T14:18:26.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 171798691847' 2024-09-13T14:18:26.263 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:18:26.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691847 -lt 171798691847 2024-09-13T14:18:26.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:18:26.631 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509465 2024-09-13T14:18:26.631 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:18:26.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:18:26.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509465 2024-09-13T14:18:26.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:18:26.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509465 2024-09-13T14:18:26.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-09-13T14:18:26.635 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509465 2024-09-13T14:18:26.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:18:26.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509465 -lt 64424509465 2024-09-13T14:18:26.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-09-13T14:18:26.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-09-13T14:18:27.254 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T14:18:27.254 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-09-13T14:18:16.919637+0000 21'10 41:105 [1,0] 1 [1,0] 1 21'10 2024-09-13T14:18:16.919425+0000 21'10 2024-09-13T14:18:16.919425+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:18:16.919425+0000 10 0 2024-09-13T14:18:27.254 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T14:18:27.254 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T14:18:27.255 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:18:27.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-09-13T14:18:27.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-09-13T14:18:27.562 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:18:27.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T14:18:27.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:18:27.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:18:27.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:18:27.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:18:27.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:18:27.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:18:27.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:18:27.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:18:27.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:18:27.689 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:18:27.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:18:27.690 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:18:27.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:18:27.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:18:27.692 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:18:27.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:18:27.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:18:27.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:18:27.694 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:18:27.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:18:27.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:18:27.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:18:27.711 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:18:27.712 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:18:27.712 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:18:27.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:18:27.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:18:27.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:18:27.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T14:18:27.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T14:18:27.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T14:18:27.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T14:18:27.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:18:27.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:18:27.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:18:27.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:18:27.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:18:27.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:18:27.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:18:27.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:18:27.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:18:27.716 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:18:27.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:18:27.717 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:18:27.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:18:27.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:18:27.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:18:27.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:18:27.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:18:27.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:18:27.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:18:27.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:18:27.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:18:27.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:18:27.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:18:27.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:18:27.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:18:27.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:18:27.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:18:27.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:18:27.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T14:18:27.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:18:27.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:18:27.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:18:27.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T14:18:27.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:18:27.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T14:18:27.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:18:27.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T14:18:27.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-09-13T14:18:27.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-09-13T14:18:27.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-09-13T14:18:27.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-09-13T14:18:27.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T14:18:27.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T14:18:27.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T14:18:27.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T14:18:27.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T14:18:27.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T14:18:28.194 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T14:18:28.194 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:18:28.194 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:18:28.194 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:18:28.194 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:18:28.194 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:18:28.195 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:18:28.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T14:18:28.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T14:18:28.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T14:18:28.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:18:28.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:18:28.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T14:18:28.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T14:18:28.417 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:18:28.417 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:18:28.417 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:18:28.418 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:18:28.418 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:18:28.418 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:18:28.418 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:18:28.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:18:28.419 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T14:18:28.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T14:18:28.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:18:28.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:18:28.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T14:18:28.536 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:18:28.536 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T14:18:28.536 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:18:28.536 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:18:28.536 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:18:28.536 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:18:28.536 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:18:28.537 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:18:28.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:18:28.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T14:18:28.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-09-13T14:18:28.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T14:18:28.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T14:18:28.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T14:18:28.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T14:18:28.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T14:18:28.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T14:18:29.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T14:18:29.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:18:29.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:18:29.531 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:18:29.532 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:18:29.532 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:18:29.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:18:29.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:18:29.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:18:29.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-13T14:18:29.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-09-13T14:18:29.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-09-13T14:18:29.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-13T14:18:29.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:18:29.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:18:29.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T14:18:29.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:18:29.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:18:29.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:18:29.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:18:29.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:18:29.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:18:29.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:18:29.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:18:29.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:18:29.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:18:29.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:18:29.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:18:29.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:18:29.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:18:29.586 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:18:29.586 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:18:29.586 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:18:29.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:18:29.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:18:29.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:18:29.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:18:29.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:18:29.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:18:29.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:18:29.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:18:29.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:18:29.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:18:29.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:18:29.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:18:29.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-13T14:18:29.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:18:29.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:18:29.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=16f18aef-79ba-4a59-b049-915eae4f43ca 2024-09-13T14:18:29.590 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 16f18aef-79ba-4a59-b049-915eae4f43ca 2024-09-13T14:18:29.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 16f18aef-79ba-4a59-b049-915eae4f43ca' 2024-09-13T14:18:29.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:18:29.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC1SeRmHayxJBAAf/ciOy0idj9yW+hCzjAHig== 2024-09-13T14:18:29.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC1SeRmHayxJBAAf/ciOy0idj9yW+hCzjAHig=="}' 2024-09-13T14:18:29.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 16f18aef-79ba-4a59-b049-915eae4f43ca -i td/osd-scrub-repair/0/new.json 2024-09-13T14:18:29.859 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:18:29.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T14:18:29.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQC1SeRmHayxJBAAf/ciOy0idj9yW+hCzjAHig== --osd-uuid 16f18aef-79ba-4a59-b049-915eae4f43ca 2024-09-13T14:18:29.911 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:18:29.912+0000 7f49ef3c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:18:29.926 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:18:29.928+0000 7f49ef3c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:18:29.927 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:18:29.928+0000 7f49ef3c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:18:29.928 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:18:29.932+0000 7f49ef3c77c0 -1 bdev(0x55c35f24dc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:18:29.928 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:18:29.932+0000 7f49ef3c77c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T14:18:36.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T14:18:36.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:18:36.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T14:18:36.331 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:18:36.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:18:36.751 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:18:36.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T14:18:36.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-13T14:18:36.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:18:36.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:18:36.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:18:36.780 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:18:36.788+0000 7f791c80a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:18:36.783 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:18:36.788+0000 7f791c80a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:18:36.784 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:18:36.792+0000 7f791c80a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:18:37.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T14:18:37.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:18:37.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:18:37.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:18:37.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:18:37.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:37.097 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:18:37.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:18:37.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:37.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:37.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:38.266 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:18:38.272+0000 7f791c80a7c0 -1 Falling back to public interface 2024-09-13T14:18:38.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:38.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:38.434 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:18:38.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:18:38.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:38.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:38.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:39.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:39.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:39.751 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:18:39.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:18:39.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:39.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:40.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:41.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:41.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:41.071 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:18:41.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:18:41.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:41.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:41.226 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:18:41.232+0000 7f791c80a7c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:18:41.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:42.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:42.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:42.407 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:18:42.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:18:42.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:42.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:42.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:43.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:43.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:43.790 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:18:43.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:18:43.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:43.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:44.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:45.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:45.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:45.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:18:45.230 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:18:45.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:45.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:45.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:46.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:46.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:46.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:18:46.570 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:18:46.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:46.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:47.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:48.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:48.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:48.068 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:18:48.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:18:48.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:48.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:48.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:49.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:49.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:49.354 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:18:49.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:18:49.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:49.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:49.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:50.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:50.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:50.691 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:18:50.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:18:50.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:50.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:51.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:52.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:52.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:52.029 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:18:52.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:18:52.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:52.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:52.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:53.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:53.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:53.350 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:18:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:18:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:53.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:54.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:54.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:54.674 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:18:54.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:18:54.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:54.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:55.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:56.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:56.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:56.457 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:18:56.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:18:56.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:56.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:57.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:58.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:58.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:58.221 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:18:58.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:18:58.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:58.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:18:58.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:18:59.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:18:59.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:18:59.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:18:59.965 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:18:59.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:18:59.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:19:00.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:01.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:01.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:01.574 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:19:01.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:19:01.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:01.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:19:02.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:03.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:03.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:03.195 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:19:03.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:19:03.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:03.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:19:03.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:04.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:04.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:04.706 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:19:04.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:19:04.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:04.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:19:05.084 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1725934235,v1:127.0.0.1:6803/1725934235] [v2:127.0.0.1:6804/1725934235,v1:127.0.0.1:6805/1725934235] exists,up 16f18aef-79ba-4a59-b049-915eae4f43ca 2024-09-13T14:19:05.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:19:05.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:19:05.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:19:05.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-09-13T14:19:05.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-13T14:19:05.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:19:05.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:19:05.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T14:19:05.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:19:05.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:19:05.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:19:05.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:19:05.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:19:05.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:19:05.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:19:05.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:19:05.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:19:05.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:19:05.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:19:05.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:19:05.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:19:05.088 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:19:05.088 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:19:05.088 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:19:05.088 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:19:05.088 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:19:05.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:19:05.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:19:05.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:19:05.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:19:05.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:19:05.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:19:05.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:19:05.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:19:05.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:19:05.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:19:05.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:19:05.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-13T14:19:05.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:19:05.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:19:05.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6ac5fdf7-e869-47f5-b73d-120b6c5cd106 2024-09-13T14:19:05.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 6ac5fdf7-e869-47f5-b73d-120b6c5cd106' 2024-09-13T14:19:05.090 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 6ac5fdf7-e869-47f5-b73d-120b6c5cd106 2024-09-13T14:19:05.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:19:05.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDZSeRmTqO/BhAA0QtQgeAkVx9+LIjoK87ahA== 2024-09-13T14:19:05.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDZSeRmTqO/BhAA0QtQgeAkVx9+LIjoK87ahA=="}' 2024-09-13T14:19:05.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6ac5fdf7-e869-47f5-b73d-120b6c5cd106 -i td/osd-scrub-repair/1/new.json 2024-09-13T14:19:05.505 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:19:05.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T14:19:05.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDZSeRmTqO/BhAA0QtQgeAkVx9+LIjoK87ahA== --osd-uuid 6ac5fdf7-e869-47f5-b73d-120b6c5cd106 2024-09-13T14:19:05.564 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:05.571+0000 7f00052d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:19:05.599 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:05.603+0000 7f00052d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:19:05.601 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:05.607+0000 7f00052d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:19:05.601 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:05.607+0000 7f00052d97c0 -1 bdev(0x5597341bfc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:19:05.601 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:05.607+0000 7f00052d97c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T14:19:12.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T14:19:12.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:19:12.129 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:19:12.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T14:19:12.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:19:12.535 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:19:12.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T14:19:12.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:19:12.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:19:12.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:19:12.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-13T14:19:12.571 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:12.574+0000 7f25fb44a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:19:12.603 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:12.606+0000 7f25fb44a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:19:12.605 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:12.610+0000 7f25fb44a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:19:12.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T14:19:12.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:19:12.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:19:12.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:19:12.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:19:12.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:12.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:19:12.888 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:19:12.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:12.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:13.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:13.655 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:13.658+0000 7f25fb44a7c0 -1 Falling back to public interface 2024-09-13T14:19:14.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:14.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:14.210 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:19:14.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:19:14.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:14.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:14.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:15.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:15.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:15.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:19:15.530 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:19:15.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:15.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:15.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:16.436 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:16.438+0000 7f25fb44a7c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:19:16.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:16.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:16.867 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:19:16.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:19:16.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:16.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:17.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:18.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:18.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:18.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:19:18.250 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:19:18.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:18.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:18.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:19.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:19.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:19.578 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:19:19.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:19:19.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:19.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:19.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:20.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:20.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:20.903 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:19:20.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:19:20.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:20.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:21.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:22.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:22.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:22.412 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:19:22.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:19:22.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:22.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:22.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:23.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:23.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:23.741 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:19:23.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:19:23.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:23.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:24.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:25.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:25.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:25.075 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:19:25.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:19:25.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:25.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:25.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:26.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:26.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:26.396 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:19:26.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:19:26.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:26.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:26.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:27.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:27.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:27.740 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:19:27.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:19:27.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:27.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:28.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:29.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:29.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:29.061 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:19:29.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:19:29.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:29.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:29.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:30.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:30.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:30.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:19:30.401 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:19:30.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:30.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:31.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:32.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:32.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:32.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:19:32.301 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:19:32.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:32.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:32.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:33.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:33.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:33.835 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:19:33.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:19:33.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:33.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:34.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:35.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:35.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:35.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:19:35.619 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:19:35.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:35.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:37.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:37.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:37.129 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:19:37.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:19:37.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:37.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:37.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:38.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:38.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:38.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:19:38.538 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:19:38.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:38.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:39.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:40.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:40.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:40.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:19:40.205 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:19:40.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:40.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:40.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:41.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:41.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:41.720 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T14:19:41.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T14:19:41.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:41.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:19:42.057 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4242344083,v1:127.0.0.1:6811/4242344083] [v2:127.0.0.1:6812/4242344083,v1:127.0.0.1:6813/4242344083] exists,up 6ac5fdf7-e869-47f5-b73d-120b6c5cd106 2024-09-13T14:19:42.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:19:42.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:19:42.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:19:42.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-09-13T14:19:42.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-13T14:19:42.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:19:42.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:19:42.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T14:19:42.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:19:42.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:19:42.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:19:42.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:19:42.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:19:42.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:19:42.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T14:19:42.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T14:19:42.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:19:42.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:19:42.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:19:42.060 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:19:42.060 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:19:42.060 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:19:42.062 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:19:42.062 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:19:42.062 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:19:42.062 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:19:42.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:19:42.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:19:42.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:19:42.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:19:42.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:19:42.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:19:42.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:19:42.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:19:42.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:19:42.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:19:42.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:19:42.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-13T14:19:42.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T14:19:42.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:19:42.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=af01263e-11fa-43dc-811b-f7d84be15de3 2024-09-13T14:19:42.064 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 af01263e-11fa-43dc-811b-f7d84be15de3 2024-09-13T14:19:42.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 af01263e-11fa-43dc-811b-f7d84be15de3' 2024-09-13T14:19:42.064 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:19:42.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD+SeRm6N+IBRAAs3HhmDNrjcB3qZjaU5nzXw== 2024-09-13T14:19:42.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD+SeRm6N+IBRAAs3HhmDNrjcB3qZjaU5nzXw=="}' 2024-09-13T14:19:42.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new af01263e-11fa-43dc-811b-f7d84be15de3 -i td/osd-scrub-repair/2/new.json 2024-09-13T14:19:42.526 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:19:42.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T14:19:42.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQD+SeRm6N+IBRAAs3HhmDNrjcB3qZjaU5nzXw== --osd-uuid af01263e-11fa-43dc-811b-f7d84be15de3 2024-09-13T14:19:42.588 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:42.589+0000 7f14308017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:19:42.611 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:42.613+0000 7f14308017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:19:42.612 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:42.613+0000 7f14308017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:19:42.612 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:42.617+0000 7f14308017c0 -1 bdev(0x556c581afc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T14:19:42.612 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:42.617+0000 7f14308017c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T14:19:49.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T14:19:49.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:19:49.060 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T14:19:49.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T14:19:49.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:19:49.462 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T14:19:49.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T14:19:49.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:19:49.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:19:49.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:19:49.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-13T14:19:49.499 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:49.501+0000 7fc337bff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:19:49.533 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:49.537+0000 7fc337bff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:19:49.534 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:49.537+0000 7fc337bff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:19:49.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T14:19:49.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:19:49.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:19:49.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:19:49.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:19:49.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:49.816 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:19:49.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:19:49.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:49.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:19:50.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:51.051 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:51.053+0000 7fc337bff7c0 -1 Falling back to public interface 2024-09-13T14:19:51.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:51.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:51.177 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:19:51.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:19:51.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:51.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:19:51.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:52.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:52.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:52.507 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:19:52.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:19:52.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:52.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:19:52.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:53.762 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:19:53.765+0000 7fc337bff7c0 -1 osd.2 0 log_to_monitors true 2024-09-13T14:19:53.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:53.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:53.845 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:19:53.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:19:53.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:53.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:19:54.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:55.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:55.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:55.273 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:19:55.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:19:55.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:55.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:19:55.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:56.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:56.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:56.627 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:19:56.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:19:56.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:56.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:19:57.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:58.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:58.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:58.089 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:19:58.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:19:58.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:58.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:19:58.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:19:59.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:19:59.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:19:59.413 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:19:59.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:19:59.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:19:59.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:19:59.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:00.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:00.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:00.790 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:20:00.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:20:00.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:00.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:20:01.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:02.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:02.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:02.235 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:20:02.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:20:02.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:02.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:20:02.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:03.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:03.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:03.573 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:20:03.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:20:03.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:03.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:20:03.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:04.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:04.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:04.910 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:20:04.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:20:04.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:04.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:20:05.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:06.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:06.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:06.279 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:20:06.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:20:06.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:06.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:20:06.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:07.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:07.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:07.617 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:20:07.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:20:07.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:07.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:20:08.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:09.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:09.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:09.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:20:09.095 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:20:09.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:09.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:20:09.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:10.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:10.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:10.877 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:20:10.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:20:10.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:10.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:20:11.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:12.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:12.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:12.647 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:20:12.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:20:12.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:12.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:20:13.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:14.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:14.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:14.401 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:20:14.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:20:14.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:14.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:20:14.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:15.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:15.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:15.843 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:20:15.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:20:15.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:15.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:20:16.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:17.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:17.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:17.687 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:20:17.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:20:17.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:17.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:20:18.053 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1757636172,v1:127.0.0.1:6819/1757636172] [v2:127.0.0.1:6820/1757636172,v1:127.0.0.1:6821/1757636172] exists,up af01263e-11fa-43dc-811b-f7d84be15de3 2024-09-13T14:20:18.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:20:18.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:20:18.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:20:18.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-09-13T14:20:18.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-13T14:20:18.479 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'testpool' created 2024-09-13T14:20:18.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:20:19.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-09-13T14:20:20.327 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 1 size to 2 2024-09-13T14:20:20.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-09-13T14:20:20.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:20:20.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:20:20.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:20:20.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:20:20.352 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:20:20.352 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:20:20.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:20:20.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:20:20.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:20:20.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:20:20.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:20:20.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:20:20.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:20:20.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:20:20.471 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:20:20.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:20:20.919 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:20:20.919 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:20:20.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:20:20.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:20:20.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:20:21.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836500 2024-09-13T14:20:21.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836500 2024-09-13T14:20:21.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500' 2024-09-13T14:20:21.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:20:21.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:20:21.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-13T14:20:21.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-13T14:20:21.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672974' 2024-09-13T14:20:21.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:20:21.293 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:20:21.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509447 2024-09-13T14:20:21.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509447 2024-09-13T14:20:21.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672974 2-64424509447' 2024-09-13T14:20:21.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:20:21.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836500 2024-09-13T14:20:21.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:20:21.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:20:21.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836500 2024-09-13T14:20:21.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:20:21.438 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836500 2024-09-13T14:20:21.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836500 2024-09-13T14:20:21.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-13T14:20:21.439 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:20:21.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836498 -lt 21474836500 2024-09-13T14:20:21.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:20:22.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:20:22.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:20:23.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-13T14:20:23.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:20:23.083 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-13T14:20:23.083 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:20:23.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:20:23.084 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-13T14:20:23.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:20:23.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-13T14:20:23.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-13T14:20:23.086 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672974 2024-09-13T14:20:23.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:20:23.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672974 2024-09-13T14:20:23.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:20:23.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509447 2024-09-13T14:20:23.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:20:23.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:20:23.410 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509447 2024-09-13T14:20:23.410 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:20:23.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509447 2024-09-13T14:20:23.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-13T14:20:23.411 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509447 2024-09-13T14:20:23.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:20:23.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-13T14:20:23.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:20:23.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:20:23.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:20:24.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:20:24.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:20:24.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:20:24.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:20:24.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:20:24.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:20:24.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:20:24.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:20:24.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:20:24.521 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:20:24.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:20:24.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:20:24.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:20:24.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:20:24.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:20:24.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-09-13T14:20:24.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-09-13T14:20:24.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-13T14:20:25.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-09-13T14:20:25.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-13T14:20:25.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:20:25.024 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-13T14:20:25.024 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:20:25.025 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:20:25.025 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:20:25.025 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:20:25.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T14:20:25.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T14:20:25.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:20:25.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T14:20:25.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:20:25.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:20:25.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T14:20:25.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:20:25.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T14:20:25.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:20:25.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:20:25.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T14:20:25.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:20:25.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T14:20:25.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:20:25.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:20:25.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:20:25.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:20:25.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:20:26.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:20:26.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T14:20:26.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:20:26.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:20:26.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T14:20:26.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:20:26.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:20:26.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-13T14:20:27.723 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-13T14:20:28.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-13T14:20:28.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:20:28.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:20:28.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T14:20:28.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:20:28.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:20:28.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:20:28.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:20:28.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:20:28.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:20:28.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:20:28.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:20:28.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:20:28.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:20:28.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:20:28.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:20:28.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:20:28.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:20:28.267 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:20:28.268 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:20:28.268 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:20:28.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:20:28.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:20:28.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:20:28.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:20:28.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:20:28.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:20:28.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:20:28.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:20:28.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:20:28.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:20:28.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-13T14:20:28.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:20:28.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T14:20:28.269 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T14:20:28.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-13T14:20:28.270 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T14:20:28.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T14:20:28.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:20:28.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:20:28.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:20:28.307 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:20:28.312+0000 7fb8543ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:20:28.314 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:20:28.320+0000 7fb8543ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:20:28.315 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:20:28.320+0000 7fb8543ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:20:28.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T14:20:28.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:20:28.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:20:28.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:20:28.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:20:28.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:28.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:20:28.604 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:20:28.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:28.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:20:28.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:29.568 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:20:29.572+0000 7fb8543ba7c0 -1 Falling back to public interface 2024-09-13T14:20:29.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:29.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:29.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:20:29.965 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:20:29.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:29.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:20:30.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:31.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:31.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:31.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:20:31.300 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:20:31.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:31.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:20:31.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:32.426 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:20:32.432+0000 7fb8543ba7c0 -1 osd.0 21 log_to_monitors true 2024-09-13T14:20:32.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:32.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:32.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:20:32.628 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:20:32.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:32.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:20:32.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:20:33.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:20:33.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:33.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:20:33.962 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:20:33.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:33.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:20:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/593780900,v1:127.0.0.1:6803/593780900] [v2:127.0.0.1:6804/593780900,v1:127.0.0.1:6805/593780900] exists,up 16f18aef-79ba-4a59-b049-915eae4f43ca 2024-09-13T14:20:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:20:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:20:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:20:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:20:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:20:34.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:20:34.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:20:34.303 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:20:34.303 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:20:34.303 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:20:34.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:20:34.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:20:34.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:20:34.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:20:34.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:20:34.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:20:34.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:20:34.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:20:34.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:20:34.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:20:34.785 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:20:34.785 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:20:34.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:20:34.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:20:34.786 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:20:34.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182402 2024-09-13T14:20:34.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182402 2024-09-13T14:20:34.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402' 2024-09-13T14:20:34.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:20:34.950 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:20:35.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672978 2024-09-13T14:20:35.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672978 2024-09-13T14:20:35.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672978' 2024-09-13T14:20:35.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:20:35.095 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:20:35.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-13T14:20:35.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-13T14:20:35.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672978 2-64424509451' 2024-09-13T14:20:35.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:20:35.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182402 2024-09-13T14:20:35.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:20:35.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:20:35.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182402 2024-09-13T14:20:35.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:20:35.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182402 2024-09-13T14:20:35.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-13T14:20:35.240 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 107374182402 2024-09-13T14:20:35.240 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:20:35.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 107374182402 2024-09-13T14:20:35.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:20:36.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:20:36.556 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:20:36.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-13T14:20:36.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:20:36.894 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672978 2024-09-13T14:20:36.894 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:20:36.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:20:36.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672978 2024-09-13T14:20:36.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:20:36.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672978 2024-09-13T14:20:36.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-13T14:20:36.898 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672978 2024-09-13T14:20:36.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:20:37.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-13T14:20:37.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:20:37.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-13T14:20:37.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:20:37.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:20:37.247 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-13T14:20:37.247 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:20:37.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-13T14:20:37.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-13T14:20:37.249 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509451 2024-09-13T14:20:37.249 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:20:37.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-13T14:20:37.581 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:20:37.581 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:20:37.581 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:20:37.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:20:37.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:20:37.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:20:37.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:20:37.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:20:37.992 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:20:37.992 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:20:37.992 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:20:38.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:20:38.311 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:20:38.311 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:20:38.311 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:20:38.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:20:38.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:20:38.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:20:38.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-09-13T14:20:38.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-13T14:20:38.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:20:38.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2024-09-13T14:20:38.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:20:39.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-09-13T14:20:39.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-09-13T14:20:39.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:20:39.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:20:39.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:20:39.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:20:39.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-09-13T14:20:39.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-09-13T14:20:39.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:20:39.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:20:39.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:20:39.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:20:39.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-09-13T14:20:18.432765+0000 2024-09-13T14:20:39.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-09-13T14:20:39.792 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T14:20:39.793 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T14:20:39.793 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T14:20:39.793 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-06T14:18:59.799832+0000" 2024-09-13T14:20:39.793 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T14:20:39.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-09-13T14:20:18.432765+0000 2024-09-13T14:20:39.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T14:20:39.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:20:18.432765+0000 2024-09-13T14:20:39.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:20:39.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:20:39.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:20:39.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:20:39.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:20:39.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:20:39.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:20:39.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:20:40.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:20:18.432765+0000 '>' 2024-09-13T14:20:18.432765+0000 2024-09-13T14:20:40.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:20:41.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:20:41.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:20:41.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:20:41.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:20:41.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:20:41.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:20:41.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:20:41.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:20:18.432765+0000 '>' 2024-09-13T14:20:18.432765+0000 2024-09-13T14:20:41.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:20:42.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:20:42.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:20:42.427 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T14:20:42.427 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:20:42.427 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:20:42.427 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:20:42.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:20:42.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:20:41.288199+0000 '>' 2024-09-13T14:20:18.432765+0000 2024-09-13T14:20:42.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:20:42.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-13T14:20:43.061 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T14:20:43.061 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-13T14:20:41.461477+0000 21'1 26:67 [1,0] 1 [1,0] 1 21'1 2024-09-13T14:20:41.288199+0000 21'1 2024-09-13T14:20:41.288199+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:20:41.288199+0000 1 0 2024-09-13T14:20:43.061 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T14:20:43.061 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T14:20:43.062 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:20:43.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-09-13T14:20:45.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-13T14:20:45.395 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T14:20:45.395 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-13T14:20:41.461477+0000 21'1 26:67 [1,0] 1 [1,0] 1 21'1 2024-09-13T14:20:41.288199+0000 21'1 2024-09-13T14:20:41.288199+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:20:41.288199+0000 1 0 2024-09-13T14:20:45.395 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T14:20:45.395 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T14:20:45.395 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:20:45.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-09-13T14:20:47.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-13T14:20:47.727 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T14:20:47.727 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-13T14:20:41.461477+0000 21'1 26:67 [1,0] 1 [1,0] 1 21'1 2024-09-13T14:20:41.288199+0000 21'1 2024-09-13T14:20:41.288199+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:20:41.288199+0000 1 0 2024-09-13T14:20:47.728 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T14:20:47.728 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T14:20:47.728 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:20:47.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-09-13T14:20:52.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-09-13T14:20:52.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:20:52.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:20:52.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:20:52.747 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:20:52.749 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:20:52.749 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:20:52.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:20:52.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:20:52.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:20:52.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:20:52.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:20:52.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:20:52.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:20:52.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:20:52.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:20:53.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:20:53.207 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:20:53.207 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:20:53.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:20:53.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:20:53.208 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:20:53.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182407 2024-09-13T14:20:53.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182407 2024-09-13T14:20:53.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182407' 2024-09-13T14:20:53.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:20:53.371 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:20:53.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672983 2024-09-13T14:20:53.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672983 2024-09-13T14:20:53.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182407 1-42949672983' 2024-09-13T14:20:53.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:20:53.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:20:53.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509456 2024-09-13T14:20:53.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509456 2024-09-13T14:20:53.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182407 1-42949672983 2-64424509456' 2024-09-13T14:20:53.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:20:53.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182407 2024-09-13T14:20:53.656 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:20:53.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:20:53.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182407 2024-09-13T14:20:53.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:20:53.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182407 2024-09-13T14:20:53.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-09-13T14:20:53.658 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 107374182407 2024-09-13T14:20:53.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:20:53.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182405 -lt 107374182407 2024-09-13T14:20:53.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:20:54.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:20:54.950 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:20:55.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182407 -lt 107374182407 2024-09-13T14:20:55.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:20:55.278 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:20:55.278 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672983 2024-09-13T14:20:55.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:20:55.280 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672983 2024-09-13T14:20:55.280 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:20:55.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672983 2024-09-13T14:20:55.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-13T14:20:55.281 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672983 2024-09-13T14:20:55.281 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:20:55.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-13T14:20:55.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:20:55.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:20:55.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509456 2024-09-13T14:20:55.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:20:55.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509456 2024-09-13T14:20:55.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:20:55.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509456 2024-09-13T14:20:55.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-13T14:20:55.644 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509456 2024-09-13T14:20:55.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:20:55.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-13T14:20:55.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:20:55.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:20:55.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:20:56.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:20:56.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:20:56.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:20:56.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:20:56.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:20:56.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:20:56.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:20:56.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:20:56.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:20:56.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:20:56.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:20:56.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:20:57.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:20:57.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:20:57.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:20:57.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-13T14:20:57.458 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T14:20:57.458 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-13T14:20:41.461477+0000 21'1 26:67 [1,0] 1 [1,0] 1 21'1 2024-09-13T14:20:41.288199+0000 21'1 2024-09-13T14:20:41.288199+0000 0 1 periodic scrub scheduled @ 2024-09-14T14:20:41.288199+0000 1 0 2024-09-13T14:20:57.458 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T14:20:57.458 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T14:20:57.458 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:20:57.475 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-09-13T14:20:57.475 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-13T14:20:57.476 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:20:57.476 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-13T14:20:57.476 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:20:57.476 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:20:57.477 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:20:57.477 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:20:57.810 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T14:20:57.811 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:20:57.811 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T14:20:58.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-13T14:20:58.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:20:58.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:20:58.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T14:20:58.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:20:58.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-13T14:20:58.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:20:58.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:20:58.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T14:20:58.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:20:58.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T14:20:58.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:20:58.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:20:58.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:20:58.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:20:58.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:20:58.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:20:58.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-13T14:20:58.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:20:58.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:20:58.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T14:20:58.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:20:58.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:20:58.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-13T14:20:59.000 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:20:59.000 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:20:59.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-13T14:20:59.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:20:59.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:20:59.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T14:20:59.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:20:59.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:20:59.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:20:59.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:20:59.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:20:59.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:20:59.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:20:59.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:20:59.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:20:59.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:20:59.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:20:59.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:20:59.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:20:59.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:20:59.291 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:20:59.292 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:20:59.292 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:20:59.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:20:59.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:20:59.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:20:59.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:20:59.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:20:59.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:20:59.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:20:59.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:20:59.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:20:59.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:20:59.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-13T14:20:59.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:20:59.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T14:20:59.293 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T14:20:59.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-13T14:20:59.293 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T14:20:59.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T14:20:59.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:20:59.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:20:59.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:20:59.336 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:20:59.339+0000 7fc855dbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:20:59.339 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:20:59.343+0000 7fc855dbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:20:59.341 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:20:59.343+0000 7fc855dbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:20:59.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T14:20:59.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:20:59.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:20:59.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:20:59.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:20:59.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:20:59.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:20:59.631 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:20:59.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:20:59.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:21:00.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:21:00.448 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:00.450+0000 7fc855dbe7c0 -1 Falling back to public interface 2024-09-13T14:21:01.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:21:01.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:01.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:21:01.006 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:21:01.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:01.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:21:01.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:21:02.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:21:02.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:02.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:21:02.344 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:21:02.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:02.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:21:02.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:21:03.502 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:03.506+0000 7fc855dbe7c0 -1 osd.0 26 log_to_monitors true 2024-09-13T14:21:03.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:21:03.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:03.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:21:03.676 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:21:03.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:03.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:21:04.019 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/114114832,v1:127.0.0.1:6803/114114832] [v2:127.0.0.1:6804/114114832,v1:127.0.0.1:6805/114114832] exists,up 16f18aef-79ba-4a59-b049-915eae4f43ca 2024-09-13T14:21:04.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:21:04.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:21:04.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:21:04.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:21:04.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:21:04.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:21:04.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:21:04.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:21:04.020 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:21:04.020 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:21:04.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:21:04.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:21:04.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:21:04.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:21:04.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:21:04.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:21:04.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:21:04.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:21:04.121 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:21:04.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:21:04.610 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:21:04.611 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:21:04.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:21:04.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:21:04.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:21:04.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018882 2024-09-13T14:21:04.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018882 2024-09-13T14:21:04.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882' 2024-09-13T14:21:04.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:21:04.792 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:21:04.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672986 2024-09-13T14:21:04.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672986 2024-09-13T14:21:04.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672986' 2024-09-13T14:21:04.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:21:04.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:21:05.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-13T14:21:05.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-13T14:21:05.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672986 2-64424509459' 2024-09-13T14:21:05.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:21:05.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018882 2024-09-13T14:21:05.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:21:05.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:21:05.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018882 2024-09-13T14:21:05.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:21:05.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018882 2024-09-13T14:21:05.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-09-13T14:21:05.115 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 128849018882 2024-09-13T14:21:05.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:21:05.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 128849018882 2024-09-13T14:21:05.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:21:06.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:21:06.442 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:21:06.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-13T14:21:06.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:21:06.792 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672986 2024-09-13T14:21:06.793 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:21:06.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:21:06.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672986 2024-09-13T14:21:06.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:21:06.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672986 2024-09-13T14:21:06.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-09-13T14:21:06.797 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672986 2024-09-13T14:21:06.797 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:21:07.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672986 -lt 42949672986 2024-09-13T14:21:07.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:21:07.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-13T14:21:07.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:21:07.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:21:07.127 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-13T14:21:07.127 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:21:07.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-13T14:21:07.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-13T14:21:07.129 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509459 2024-09-13T14:21:07.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:21:07.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-13T14:21:07.461 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:21:07.462 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:21:07.462 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:21:07.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:21:07.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:21:07.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:21:07.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:21:07.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:21:07.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:21:07.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:21:07.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:21:08.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:21:08.198 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:21:08.198 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:21:08.198 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:21:08.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:21:08.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:21:08.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:21:08.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:21:08.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:21:08.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-13T14:21:08.626 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:20:40.251+0000 7f25df3d4640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-09-13T14:21:08.626 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:20:40.251+0000 7f25df3d4640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-09-13T14:21:08.627 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:20:40.251+0000 7f25df3d4640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-09-13T14:21:08.627 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:20:41.283+0000 7f25df3d4640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] NEED_AUTO] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-13T14:21:08.627 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:20:41.283+0000 7f25df3d4640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-13T14:21:08.627 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:20:41.283+0000 7f25df3d4640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-13T14:21:08.627 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:20:41.283+0000 7f25df3d4640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-09-13T14:21:08.627 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:20:41.283+0000 7f25df3d4640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-13T14:21:08.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-09-13T14:21:08.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-09-13T14:21:08.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-09-13T14:21:08.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-09-13T14:21:08.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T14:21:08.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:21:08.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:21:08.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:21:08.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:21:08.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:21:08.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:21:08.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:21:08.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:21:09.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:21:09.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:21:09.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:21:09.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:21:09.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:21:09.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:21:09.075 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:21:09.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:21:09.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:21:09.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:21:09.077 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:21:09.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:21:09.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:21:09.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:21:09.099 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:21:09.099 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:21:09.099 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:21:09.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:21:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:21:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:21:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T14:21:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T14:21:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T14:21:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T14:21:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:21:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:21:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:21:09.102 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:21:09.103 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:21:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:21:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:21:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:21:09.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:21:09.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:21:09.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:21:09.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:21:09.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:21:09.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:21:09.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:21:09.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:21:09.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:21:09.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:21:09.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:21:09.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:21:09.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:21:09.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:21:09.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:21:09.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:21:09.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:21:09.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:21:09.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:21:09.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:21:09.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T14:21:09.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:21:09.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:21:09.117 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:21:09.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T14:21:09.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:21:09.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T14:21:09.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:21:09.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T14:21:09.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-09-13T14:21:09.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-09-13T14:21:09.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-09-13T14:21:09.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-09-13T14:21:09.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T14:21:09.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T14:21:09.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T14:21:09.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T14:21:09.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T14:21:09.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T14:21:09.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T14:21:09.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:21:09.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:21:09.611 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:21:09.611 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:21:09.611 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:21:09.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:21:09.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T14:21:09.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T14:21:09.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T14:21:09.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:21:09.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:21:09.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T14:21:09.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T14:21:09.839 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:21:09.839 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:21:09.839 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:21:09.840 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:21:09.840 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:21:09.840 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:21:09.840 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:21:09.840 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:21:09.840 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T14:21:09.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T14:21:09.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:21:09.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:21:09.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T14:21:09.954 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T14:21:09.954 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:21:09.954 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:21:09.955 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:21:09.955 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:21:09.955 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:21:09.955 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:21:09.955 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:21:09.955 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:21:09.956 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T14:21:10.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-09-13T14:21:10.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T14:21:10.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T14:21:10.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T14:21:10.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T14:21:10.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T14:21:10.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T14:21:10.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T14:21:10.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:21:10.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:21:10.936 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:21:10.936 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:21:10.936 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:21:10.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:21:10.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:21:10.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:21:10.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-13T14:21:10.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-09-13T14:21:10.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-09-13T14:21:10.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-13T14:21:10.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:21:10.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:21:10.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T14:21:10.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:21:10.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:21:10.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:21:10.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:21:10.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:21:10.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:21:10.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:21:10.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:21:10.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:21:10.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:21:10.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:21:10.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:21:10.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:21:10.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:21:10.980 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:21:10.980 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:21:10.980 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:21:10.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:21:10.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:21:10.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:21:10.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:21:10.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:21:10.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:21:10.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:21:10.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:21:10.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:21:10.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:21:10.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:21:10.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:21:10.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-13T14:21:10.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:21:10.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:21:10.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2fc01b1d-60b5-4462-a0b4-3dcf78231e87 2024-09-13T14:21:10.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2fc01b1d-60b5-4462-a0b4-3dcf78231e87' 2024-09-13T14:21:10.981 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 2fc01b1d-60b5-4462-a0b4-3dcf78231e87 2024-09-13T14:21:10.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:21:10.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBXSuRmf2FEABAAY2LVw5aCloNYRjklxbdtcw== 2024-09-13T14:21:10.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBXSuRmf2FEABAAY2LVw5aCloNYRjklxbdtcw=="}' 2024-09-13T14:21:10.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2fc01b1d-60b5-4462-a0b4-3dcf78231e87 -i td/osd-scrub-repair/0/new.json 2024-09-13T14:21:11.281 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:21:11.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T14:21:11.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBXSuRmf2FEABAAY2LVw5aCloNYRjklxbdtcw== --osd-uuid 2fc01b1d-60b5-4462-a0b4-3dcf78231e87 2024-09-13T14:21:11.338 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:11.342+0000 7ffb72b4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:11.348 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:11.350+0000 7ffb72b4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:11.349 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:11.354+0000 7ffb72b4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:11.350 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:11.354+0000 7ffb72b4a7c0 -1 bdev(0x562e4d519c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:21:11.350 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:11.354+0000 7ffb72b4a7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T14:21:18.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T14:21:18.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:21:18.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T14:21:18.138 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:21:18.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:21:18.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T14:21:18.542 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:21:18.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-13T14:21:18.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:21:18.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:21:18.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:21:18.577 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:18.578+0000 7f073d9447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:18.603 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:18.606+0000 7f073d9447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:18.604 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:18.606+0000 7f073d9447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:18.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T14:21:18.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:21:18.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:21:18.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:21:18.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:21:18.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:18.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:21:18.890 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:21:18.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:18.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:21:19.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:21:20.100 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:20.102+0000 7f073d9447c0 -1 Falling back to public interface 2024-09-13T14:21:20.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:21:20.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:20.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:21:20.215 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:21:20.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:20.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:21:20.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:21:21.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:21:21.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:21.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:21:21.534 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:21:21.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:21.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:21:21.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:21:22.774 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:22.778+0000 7f073d9447c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:21:22.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:21:22.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:22.854 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:21:22.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:21:22.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:22.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:21:23.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:21:24.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:21:24.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:24.211 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:21:24.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:21:24.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:24.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:21:24.591 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3309590500,v1:127.0.0.1:6803/3309590500] [v2:127.0.0.1:6804/3309590500,v1:127.0.0.1:6805/3309590500] exists,up 2fc01b1d-60b5-4462-a0b4-3dcf78231e87 2024-09-13T14:21:24.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:21:24.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:21:24.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:21:24.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-09-13T14:21:24.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-13T14:21:24.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:21:24.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:21:24.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T14:21:24.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:21:24.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:21:24.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:21:24.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:21:24.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:21:24.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:21:24.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:21:24.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:21:24.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:21:24.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:21:24.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:21:24.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:21:24.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:21:24.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:21:24.595 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:21:24.595 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:21:24.595 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:21:24.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:21:24.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:21:24.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:21:24.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:21:24.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:21:24.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:21:24.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:21:24.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:21:24.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:21:24.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:21:24.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:21:24.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:21:24.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-13T14:21:24.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:21:24.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:21:24.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a29fd5e7-feeb-40d8-a0b8-be04d0899986 2024-09-13T14:21:24.598 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 a29fd5e7-feeb-40d8-a0b8-be04d0899986 2024-09-13T14:21:24.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a29fd5e7-feeb-40d8-a0b8-be04d0899986' 2024-09-13T14:21:24.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:21:24.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBkSuRm6Nf8JBAAtAe+ob+jP7VB84RWOruFCQ== 2024-09-13T14:21:24.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBkSuRm6Nf8JBAAtAe+ob+jP7VB84RWOruFCQ=="}' 2024-09-13T14:21:24.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a29fd5e7-feeb-40d8-a0b8-be04d0899986 -i td/osd-scrub-repair/1/new.json 2024-09-13T14:21:24.980 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:21:24.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T14:21:24.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBkSuRm6Nf8JBAAtAe+ob+jP7VB84RWOruFCQ== --osd-uuid a29fd5e7-feeb-40d8-a0b8-be04d0899986 2024-09-13T14:21:25.035 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:25.037+0000 7f0c498f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:25.065 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:25.065+0000 7f0c498f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:25.069+0000 7f0c498f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:25.069+0000 7f0c498f67c0 -1 bdev(0x56234a625c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:21:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:25.069+0000 7f0c498f67c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T14:21:31.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T14:21:31.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:21:31.819 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:21:31.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T14:21:31.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:21:32.214 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:21:32.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T14:21:32.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-13T14:21:32.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:21:32.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:21:32.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:21:32.249 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:32.249+0000 7f7c55dd57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:32.278 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:32.281+0000 7f7c55dd57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:32.279 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:32.281+0000 7f7c55dd57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:32.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T14:21:32.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:21:32.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:21:32.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:21:32.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:21:32.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:32.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:21:32.559 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:21:32.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:32.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:21:32.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:21:33.785 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:33.785+0000 7f7c55dd57c0 -1 Falling back to public interface 2024-09-13T14:21:33.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:21:33.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:33.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:21:33.890 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:21:33.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:33.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:21:34.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:21:35.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:21:35.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:35.220 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:21:35.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:21:35.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:35.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:21:35.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:21:36.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:21:36.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:36.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:21:36.539 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:21:36.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:36.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:21:36.779 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:36.781+0000 7f7c55dd57c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:21:36.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:21:37.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:21:37.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:37.885 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:21:37.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:21:37.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:37.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:21:38.223 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/225595405,v1:127.0.0.1:6811/225595405] [v2:127.0.0.1:6812/225595405,v1:127.0.0.1:6813/225595405] exists,up a29fd5e7-feeb-40d8-a0b8-be04d0899986 2024-09-13T14:21:38.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:21:38.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:21:38.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:21:38.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-09-13T14:21:38.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-13T14:21:38.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:21:38.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:21:38.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T14:21:38.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:21:38.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:21:38.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:21:38.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:21:38.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:21:38.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:21:38.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T14:21:38.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T14:21:38.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:21:38.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:21:38.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:21:38.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:21:38.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:21:38.228 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:21:38.228 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:21:38.228 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:21:38.228 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:21:38.228 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:21:38.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:21:38.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:21:38.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:21:38.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:21:38.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:21:38.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:21:38.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:21:38.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:21:38.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:21:38.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:21:38.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:21:38.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-13T14:21:38.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T14:21:38.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:21:38.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f930d074-b26c-4e4d-9713-540176c0a46c 2024-09-13T14:21:38.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 f930d074-b26c-4e4d-9713-540176c0a46c' 2024-09-13T14:21:38.230 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 f930d074-b26c-4e4d-9713-540176c0a46c 2024-09-13T14:21:38.230 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:21:38.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBySuRmbWNnDxAA3LA2P/MefEVkoDJVZh1QCw== 2024-09-13T14:21:38.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBySuRmbWNnDxAA3LA2P/MefEVkoDJVZh1QCw=="}' 2024-09-13T14:21:38.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f930d074-b26c-4e4d-9713-540176c0a46c -i td/osd-scrub-repair/2/new.json 2024-09-13T14:21:38.632 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:21:38.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T14:21:38.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBySuRmbWNnDxAA3LA2P/MefEVkoDJVZh1QCw== --osd-uuid f930d074-b26c-4e4d-9713-540176c0a46c 2024-09-13T14:21:38.688 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:38.689+0000 7f5f25caf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:38.716 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:38.717+0000 7f5f25caf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:38.717 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:38.717+0000 7f5f25caf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:38.718 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:38.721+0000 7f5f25caf7c0 -1 bdev(0x560c7fe9c800 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T14:21:38.718 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:38.721+0000 7f5f25caf7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T14:21:45.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T14:21:45.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:21:45.216 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T14:21:45.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T14:21:45.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:21:45.618 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T14:21:45.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T14:21:45.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:21:45.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-13T14:21:45.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:21:45.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:21:45.653 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:45.653+0000 7f8168ee27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:45.678 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:45.681+0000 7f8168ee27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:45.679 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:45.681+0000 7f8168ee27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:21:45.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T14:21:45.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:21:45.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:21:45.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:21:45.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:21:45.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:45.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:21:45.995 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:21:45.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:45.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:21:46.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:21:46.977 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:46.977+0000 7f8168ee27c0 -1 Falling back to public interface 2024-09-13T14:21:47.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:21:47.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:47.350 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:21:47.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:21:47.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:47.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:21:47.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:21:48.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:21:48.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:48.686 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:21:48.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:21:48.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:48.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:21:49.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:21:49.829 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:21:49.828+0000 7f8168ee27c0 -1 osd.2 0 log_to_monitors true 2024-09-13T14:21:50.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:21:50.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:50.024 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:21:50.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:21:50.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:50.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:21:50.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:21:51.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:21:51.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:21:51.441 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:21:51.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:21:51.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:21:51.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:21:51.768 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/553890183,v1:127.0.0.1:6819/553890183] [v2:127.0.0.1:6820/553890183,v1:127.0.0.1:6821/553890183] exists,up f930d074-b26c-4e4d-9713-540176c0a46c 2024-09-13T14:21:51.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:21:51.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:21:51.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:21:51.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-09-13T14:21:51.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-13T14:21:52.185 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'testpool' created 2024-09-13T14:21:52.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:21:53.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-09-13T14:21:53.639 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 1 size to 2 2024-09-13T14:21:53.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-09-13T14:21:53.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:21:53.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:21:53.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:21:53.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:21:53.661 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:21:53.661 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:21:53.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:21:53.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:21:53.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:21:53.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:21:53.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:21:53.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:21:53.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:21:53.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:21:53.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:21:54.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:21:54.304 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:21:54.304 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:21:54.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:21:54.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:21:54.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:21:54.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836488 2024-09-13T14:21:54.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836488 2024-09-13T14:21:54.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488' 2024-09-13T14:21:54.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:21:54.439 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:21:54.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672966 2024-09-13T14:21:54.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672966 2024-09-13T14:21:54.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488 1-42949672966' 2024-09-13T14:21:54.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:21:54.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:21:54.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509442 2024-09-13T14:21:54.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509442 2024-09-13T14:21:54.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488 1-42949672966 2-64424509442' 2024-09-13T14:21:54.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:21:54.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836488 2024-09-13T14:21:54.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:21:54.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:21:54.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836488 2024-09-13T14:21:54.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:21:54.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836488 2024-09-13T14:21:54.726 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836488 2024-09-13T14:21:54.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-13T14:21:54.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:21:55.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836488 2024-09-13T14:21:55.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:21:56.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:21:56.039 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:21:56.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836488 2024-09-13T14:21:56.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:21:56.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:21:56.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672966 2024-09-13T14:21:56.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:21:56.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672966 2024-09-13T14:21:56.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:21:56.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672966 2024-09-13T14:21:56.358 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672966 2024-09-13T14:21:56.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-09-13T14:21:56.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:21:56.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672966 -lt 42949672966 2024-09-13T14:21:56.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:21:56.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509442 2024-09-13T14:21:56.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:21:56.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:21:56.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509442 2024-09-13T14:21:56.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:21:56.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509442 2024-09-13T14:21:56.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-09-13T14:21:56.698 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509442 2024-09-13T14:21:56.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:21:57.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509442 -lt 64424509442 2024-09-13T14:21:57.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:21:57.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:21:57.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:21:57.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:21:57.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:21:57.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:21:57.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:21:57.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:21:57.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:21:57.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:21:57.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:21:57.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T14:21:57.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:21:57.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:21:57.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:21:58.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T14:21:58.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T14:21:58.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T14:21:58.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T14:21:58.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:21:58.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:21:58.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:21:58.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:21:58.258 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:21:58.258 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:21:58.258 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:21:58.258 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:21:58.258 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:21:58.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:21:58.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T14:21:58.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:21:58.569 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:21:58.569 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:21:58.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T14:21:58.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T14:21:58.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T14:21:58.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T14:21:58.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T14:21:58.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T14:21:58.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T14:21:58.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T14:21:58.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T14:21:59.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T14:21:59.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T14:21:59.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T14:21:59.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:21:59.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T14:21:59.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:21:59.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:21:59.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:21:59.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:21:59.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:21:59.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:21:59.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:21:59.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:21:59.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:21:59.846 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:21:59.846 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:21:59.846 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:22:00.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:22:00.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:22:00.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:22:00.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-09-13T14:22:00.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-09-13T14:22:00.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-13T14:22:00.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-09-13T14:22:00.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-13T14:22:00.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:22:00.402 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-13T14:22:00.402 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:22:00.402 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:22:00.402 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:22:00.402 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:22:00.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T14:22:00.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:22:00.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T14:22:01.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T14:22:01.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:22:01.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:22:01.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T14:22:01.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:22:01.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T14:22:01.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:22:01.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:22:01.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T14:22:01.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:22:01.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T14:22:01.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:22:01.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:22:01.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:22:01.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:22:01.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:22:01.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:22:01.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T14:22:01.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:22:01.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:22:01.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T14:22:01.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:22:01.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:22:01.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-13T14:22:03.090 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-13T14:22:03.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-13T14:22:03.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:22:03.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:22:03.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T14:22:03.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:22:03.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:22:03.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:22:03.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:22:03.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:22:03.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:22:03.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:22:03.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:22:03.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:22:03.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:22:03.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:22:03.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:22:03.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:22:03.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:22:03.640 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:22:03.640 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:22:03.640 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:22:03.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:22:03.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:22:03.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:22:03.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:22:03.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:22:03.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:22:03.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:22:03.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:22:03.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:22:03.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:22:03.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-13T14:22:03.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:22:03.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T14:22:03.642 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T14:22:03.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-13T14:22:03.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T14:22:03.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T14:22:03.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:22:03.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:22:03.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:22:03.678 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:03.684+0000 7f84320f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:22:03.681 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:03.684+0000 7f84320f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:22:03.682 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:03.688+0000 7f84320f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:22:03.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T14:22:03.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:22:03.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:22:03.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:22:03.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:22:03.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:22:03.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:22:03.969 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:22:03.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:22:03.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:22:04.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:22:04.729 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:04.732+0000 7f84320f67c0 -1 Falling back to public interface 2024-09-13T14:22:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:22:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:22:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:22:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:22:05.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:22:05.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:22:05.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:22:06.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:22:06.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:22:06.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:22:06.652 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:22:06.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:22:06.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:22:06.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:22:07.492 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:07.496+0000 7f84320f67c0 -1 osd.0 21 log_to_monitors true 2024-09-13T14:22:07.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:22:07.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:22:07.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:22:07.986 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:22:07.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:22:07.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:22:08.321 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1051182951,v1:127.0.0.1:6803/1051182951] [v2:127.0.0.1:6804/1051182951,v1:127.0.0.1:6805/1051182951] exists,up 2fc01b1d-60b5-4462-a0b4-3dcf78231e87 2024-09-13T14:22:08.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:22:08.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:22:08.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:22:08.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:22:08.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:22:08.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:22:08.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:22:08.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:22:08.323 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:22:08.323 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:22:08.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:22:08.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:22:08.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:22:08.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:22:08.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:22:08.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:22:08.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:22:08.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:22:08.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:22:08.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:22:08.777 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:22:08.777 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:22:08.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:22:08.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:22:08.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:22:08.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182402 2024-09-13T14:22:08.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182402 2024-09-13T14:22:08.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402' 2024-09-13T14:22:08.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:22:08.922 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:22:09.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672970 2024-09-13T14:22:09.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672970 2024-09-13T14:22:09.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672970' 2024-09-13T14:22:09.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:22:09.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:22:09.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509446 2024-09-13T14:22:09.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509446 2024-09-13T14:22:09.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672970 2-64424509446' 2024-09-13T14:22:09.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:22:09.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182402 2024-09-13T14:22:09.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:22:09.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:22:09.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182402 2024-09-13T14:22:09.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:22:09.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182402 2024-09-13T14:22:09.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-13T14:22:09.322 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 107374182402 2024-09-13T14:22:09.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:22:09.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 107374182402 2024-09-13T14:22:09.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:22:10.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:22:10.650 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:22:10.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-13T14:22:10.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:22:10.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672970 2024-09-13T14:22:10.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:22:10.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:22:10.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672970 2024-09-13T14:22:10.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:22:10.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672970 2024-09-13T14:22:10.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-09-13T14:22:10.990 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672970 2024-09-13T14:22:10.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:22:11.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672970 -lt 42949672970 2024-09-13T14:22:11.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:22:11.336 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509446 2024-09-13T14:22:11.336 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:22:11.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:22:11.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509446 2024-09-13T14:22:11.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:22:11.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509446 2024-09-13T14:22:11.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-13T14:22:11.339 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509446 2024-09-13T14:22:11.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:22:11.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-13T14:22:11.683 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:22:11.683 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:22:11.683 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:22:12.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:22:12.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:22:12.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:22:12.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:22:12.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:22:12.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:22:12.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:22:12.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:22:12.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T14:22:12.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:22:12.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:22:12.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:22:12.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T14:22:12.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T14:22:12.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T14:22:12.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T14:22:12.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:22:12.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:22:12.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:22:12.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:22:12.908 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:22:12.908 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:22:12.908 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:22:12.908 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:22:12.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:22:12.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:22:13.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T14:22:13.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:22:13.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:22:13.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:22:13.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T14:22:13.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T14:22:13.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T14:22:13.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T14:22:13.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T14:22:13.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T14:22:13.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T14:22:13.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T14:22:13.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T14:22:14.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T14:22:14.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T14:22:14.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T14:22:14.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:22:14.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T14:22:14.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:22:14.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:22:14.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:22:14.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:22:14.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:22:14.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:22:14.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:22:14.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:22:14.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:22:14.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:22:14.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:22:14.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:22:15.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:22:15.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:22:15.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:22:15.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-09-13T14:22:15.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-13T14:22:15.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:22:15.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2024-09-13T14:22:15.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:22:15.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-09-13T14:22:15.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-09-13T14:22:15.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:22:15.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:22:15.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:22:15.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:22:15.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-09-13T14:22:15.673 INFO:tasks.workunit.client.0.smithi112.stdout:Affected PG 1.0 w/ primary 1 2024-09-13T14:22:15.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-09-13T14:22:15.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-09-13T14:22:15.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:22:15.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:22:15.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:22:15.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:22:15.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-09-13T14:21:52.131772+0000 2024-09-13T14:22:15.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-09-13T14:22:15.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-09-13T14:22:15.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-09-13T14:22:15.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-09-13T14:22:15.996 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T14:22:15.996 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:22:15.996 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T14:22:15.996 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:22:16.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-09-13T14:21:52.131772+0000 2024-09-13T14:22:16.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-09-13T14:22:16.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-13T14:22:16.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-13T14:22:16.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_sleep 2024-09-13T14:22:16.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=3.0 2024-09-13T14:22:16.350 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-13T14:22:16.350 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-13T14:22:16.350 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T14:22:16.351 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T14:22:16.351 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:22:16.351 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:22:16.351 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:22:16.351 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.1.asok 2024-09-13T14:22:16.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.63282/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-09-13T14:22:16.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-13T14:22:16.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-09-13T14:22:16.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-13T14:22:16.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-13T14:22:16.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_auto_repair 2024-09-13T14:22:16.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-13T14:22:16.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-13T14:22:16.458 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-13T14:22:16.458 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T14:22:16.458 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T14:22:16.458 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:22:16.458 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:22:16.458 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:22:16.458 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.1.asok 2024-09-13T14:22:16.459 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.63282/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-09-13T14:22:16.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-13T14:22:16.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-09-13T14:22:16.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:22:16.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:22:16.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:22:16.876 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:22:16.876 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:22:16.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:22:16.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:22:16.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:22:17.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182404 2024-09-13T14:22:17.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182404 2024-09-13T14:22:17.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182404' 2024-09-13T14:22:17.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:22:17.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:22:17.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672972 2024-09-13T14:22:17.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672972 2024-09-13T14:22:17.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182404 1-42949672972' 2024-09-13T14:22:17.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:22:17.183 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:22:17.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509449 2024-09-13T14:22:17.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509449 2024-09-13T14:22:17.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182404 1-42949672972 2-64424509449' 2024-09-13T14:22:17.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:22:17.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182404 2024-09-13T14:22:17.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:22:17.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:22:17.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182404 2024-09-13T14:22:17.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:22:17.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182404 2024-09-13T14:22:17.324 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 107374182404 2024-09-13T14:22:17.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2024-09-13T14:22:17.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:22:17.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182403 -lt 107374182404 2024-09-13T14:22:17.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:22:18.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:22:18.626 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:22:18.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182404 -lt 107374182404 2024-09-13T14:22:18.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:22:18.961 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:22:18.961 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672972 2024-09-13T14:22:18.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:22:18.963 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672972 2024-09-13T14:22:18.963 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:22:18.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672972 2024-09-13T14:22:18.964 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672972 2024-09-13T14:22:18.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-13T14:22:18.964 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:22:19.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-13T14:22:19.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:22:19.287 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509449 2024-09-13T14:22:19.287 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:22:19.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:22:19.289 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509449 2024-09-13T14:22:19.289 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:22:19.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509449 2024-09-13T14:22:19.290 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509449 2024-09-13T14:22:19.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-13T14:22:19.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:22:19.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-13T14:22:19.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-09-13T14:22:19.642 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13 14:22:19.648331538+00:00 2024-09-13T14:22:19.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-09-13T14:22:19.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T14:22:19.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T14:22:19.645 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:22:19.645 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:22:19.645 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:22:19.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.1.asok 2024-09-13T14:22:19.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.63282/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-09-13T14:22:19.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-09-13T14:22:19.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-13T14:22:19.768 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-13T14:22:19.768 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-13T14:22:19.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-13T14:22:19.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-09-13T14:22:19.919 INFO:tasks.workunit.client.0.smithi112.stdout:{"deep":true,"must":false,"stamp":"2024-08-30T14:20:39.759831+0000"}0 ) state now: "active+clean" 2024-09-13T14:22:19.919 INFO:tasks.workunit.client.0.smithi112.stdout:"active+clean" 2024-09-13T14:22:19.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-13T14:22:19.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-13T14:22:19.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-09-13T14:22:19.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-13T14:22:20.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-13T14:22:20.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-13T14:22:20.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-13T14:22:20.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-13T14:22:20.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-13T14:22:20.378 INFO:tasks.workunit.client.0.smithi112.stdout:1 ) state now: "active+clean" 2024-09-13T14:22:20.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-09-13T14:22:20.379 INFO:tasks.workunit.client.0.smithi112.stdout:"active+clean" 2024-09-13T14:22:20.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-13T14:22:20.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-13T14:22:20.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-09-13T14:22:20.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-13T14:22:20.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-13T14:22:20.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-13T14:22:20.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-13T14:22:20.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-13T14:22:20.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-13T14:22:20.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2024-09-13T14:22:20.838 INFO:tasks.workunit.client.0.smithi112.stdout:2 ) state now: "active+clean" 2024-09-13T14:22:20.838 INFO:tasks.workunit.client.0.smithi112.stdout:"active+clean" 2024-09-13T14:22:20.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-13T14:22:20.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-13T14:22:20.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-09-13T14:22:20.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-13T14:22:21.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-13T14:22:21.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-13T14:22:21.141 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-13T14:22:21.141 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-13T14:22:21.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-13T14:22:21.298 INFO:tasks.workunit.client.0.smithi112.stdout:3 ) state now: "active+clean" 2024-09-13T14:22:21.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean"' 2024-09-13T14:22:21.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-13T14:22:21.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-13T14:22:21.298 INFO:tasks.workunit.client.0.smithi112.stdout:"active+clean" 2024-09-13T14:22:21.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 3 == 4 ']' 2024-09-13T14:22:21.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-13T14:22:21.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-13T14:22:21.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-13T14:22:21.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-13T14:22:21.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-13T14:22:21.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-13T14:22:21.781 INFO:tasks.workunit.client.0.smithi112.stdout:4 ) state now: "active+clean" 2024-09-13T14:22:21.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 4 ') state now: ' '"active+clean"' 2024-09-13T14:22:21.782 INFO:tasks.workunit.client.0.smithi112.stdout:"active+clean" 2024-09-13T14:22:21.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-13T14:22:21.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-13T14:22:21.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 4 == 4 ']' 2024-09-13T14:22:21.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo 'loop --------> ' 4 2024-09-13T14:22:21.783 INFO:tasks.workunit.client.0.smithi112.stdout:loop --------> 4 2024-09-13T14:22:21.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-13T14:22:22.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-13T14:22:22.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-13T14:22:22.084 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-13T14:22:22.084 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-13T14:22:22.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-13T14:22:22.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 5 ') state now: ' '"active+clean"' 2024-09-13T14:22:22.243 INFO:tasks.workunit.client.0.smithi112.stdout:5 ) state now: "active+clean" 2024-09-13T14:22:22.243 INFO:tasks.workunit.client.0.smithi112.stdout:"active+clean" 2024-09-13T14:22:22.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-13T14:22:22.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-13T14:22:22.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 5 == 4 ']' 2024-09-13T14:22:22.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-13T14:22:22.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-13T14:22:22.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-13T14:22:22.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-13T14:22:22.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-13T14:22:22.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-13T14:22:22.703 INFO:tasks.workunit.client.0.smithi112.stdout:6 ) state now: "active+clean" 2024-09-13T14:22:22.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 6 ') state now: ' '"active+clean"' 2024-09-13T14:22:22.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-13T14:22:22.703 INFO:tasks.workunit.client.0.smithi112.stdout:"active+clean" 2024-09-13T14:22:22.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-13T14:22:22.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 6 == 4 ']' 2024-09-13T14:22:22.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-13T14:22:23.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-13T14:22:23.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-13T14:22:23.005 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-13T14:22:23.005 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-13T14:22:23.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-09-13T14:22:23.165 INFO:tasks.workunit.client.0.smithi112.stdout:7 ) state now: "active+clean+scrubbing+deep" 2024-09-13T14:22:23.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 7 ') state now: ' '"active+clean+scrubbing+deep"' 2024-09-13T14:22:23.165 INFO:tasks.workunit.client.0.smithi112.stdout:found scrub 2024-09-13T14:22:23.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-13T14:22:23.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-09-13T14:22:23.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-09-13T14:22:23.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-09-13T14:22:23.166 INFO:tasks.workunit.client.0.smithi112.stdout:initiate_and_fetch_state ret: 0 2024-09-13T14:22:23.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-09-13T14:22:23.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-09-13T14:22:23.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-13T14:22:23.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=td/osd-scrub-repair 2024-09-13T14:22:23.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_sleep 2024-09-13T14:22:23.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=0 2024-09-13T14:22:23.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-13T14:22:23.169 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.td/osd-scrub-repair 2024-09-13T14:22:23.169 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.td/osd-scrub-repair 2024-09-13T14:22:23.169 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-09-13T14:22:23.169 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:22:23.169 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:22:23.169 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:22:23.169 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.td/osd-scrub-repair.asok 2024-09-13T14:22:23.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.63282/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-09-13T14:22:23.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test == true 2024-09-13T14:22:23.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1159: test: ==: unary operator expected 2024-09-13T14:22:23.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-09-13T14:22:23.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-09-13T14:22:23.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-09-13T14:22:23.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-09-13T14:22:23.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-09-13T14:22:23.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-13T14:22:23.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-13T14:22:23.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-13T14:22:23.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-13T14:22:23.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-13T14:22:23.393 INFO:tasks.workunit.client.0.smithi112.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-13T14:22:23.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-13T14:22:23.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-09-13T14:22:23.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-13T14:22:23.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-13T14:22:23.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-13T14:22:23.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-13T14:22:23.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-13T14:22:23.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-13T14:22:23.852 INFO:tasks.workunit.client.0.smithi112.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-13T14:22:23.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-13T14:22:23.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-13T14:22:23.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-09-13T14:22:23.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-13T14:22:24.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-13T14:22:24.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-13T14:22:24.154 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-13T14:22:24.154 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-13T14:22:24.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-13T14:22:24.310 INFO:tasks.workunit.client.0.smithi112.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-13T14:22:24.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-13T14:22:24.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-13T14:22:24.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-09-13T14:22:24.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-13T14:22:24.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-13T14:22:24.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-13T14:22:24.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-13T14:22:24.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-13T14:22:24.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-13T14:22:24.767 INFO:tasks.workunit.client.0.smithi112.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-13T14:22:24.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-13T14:22:24.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-13T14:22:24.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-09-13T14:22:24.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-13T14:22:25.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-13T14:22:25.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-13T14:22:25.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-13T14:22:25.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-13T14:22:25.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-13T14:22:25.224 INFO:tasks.workunit.client.0.smithi112.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-13T14:22:25.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-13T14:22:25.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-13T14:22:25.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-09-13T14:22:25.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-09-13T14:22:25.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:22:25.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:22:25.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:22:25.533 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:22:25.533 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:22:25.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:22:25.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:22:25.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:22:25.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182407 2024-09-13T14:22:25.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182407 2024-09-13T14:22:25.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182407' 2024-09-13T14:22:25.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:22:25.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:22:25.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672975 2024-09-13T14:22:25.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672975 2024-09-13T14:22:25.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182407 1-42949672975' 2024-09-13T14:22:25.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:22:25.845 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:22:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-13T14:22:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-13T14:22:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182407 1-42949672975 2-64424509451' 2024-09-13T14:22:25.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:22:25.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182407 2024-09-13T14:22:25.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:22:25.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:22:25.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182407 2024-09-13T14:22:25.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:22:25.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182407 2024-09-13T14:22:25.991 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 107374182407 2024-09-13T14:22:25.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-09-13T14:22:25.992 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:22:26.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182406 -lt 107374182407 2024-09-13T14:22:26.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:22:27.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:22:27.334 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:22:27.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182406 -lt 107374182407 2024-09-13T14:22:27.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:22:28.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T14:22:28.656 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:22:28.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182407 -lt 107374182407 2024-09-13T14:22:28.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:22:28.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672975 2024-09-13T14:22:28.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:22:28.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:22:28.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672975 2024-09-13T14:22:28.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:22:28.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672975 2024-09-13T14:22:28.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-13T14:22:28.980 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672975 2024-09-13T14:22:28.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:22:29.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-13T14:22:29.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:22:29.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-13T14:22:29.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:22:29.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:22:29.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-13T14:22:29.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:22:29.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-13T14:22:29.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-13T14:22:29.331 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509451 2024-09-13T14:22:29.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:22:29.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509451 2024-09-13T14:22:29.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-13T14:22:29.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-13T14:22:29.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-13T14:22:29.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-13T14:22:29.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-13T14:22:30.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-09-13T14:22:30.160 INFO:tasks.workunit.client.0.smithi112.stdout:wait-scrub-end state now: "active+clean" 2024-09-13T14:22:30.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-09-13T14:22:30.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-09-13T14:22:30.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-09-13T14:22:30.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-09-13T14:22:30.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-09-13T14:22:30.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-09-13T14:22:30.461 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T14:22:30.461 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-13T14:22:26.179661+0000 21'1 26:55 [1,0] 1 [1,0] 1 21'1 2024-09-13T14:22:25.882759+0000 21'1 2024-09-13T14:22:25.882759+0000 0 4 periodic scrub scheduled @ 2024-09-14T14:22:25.882759+0000 1 0 2024-09-13T14:22:30.461 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T14:22:30.461 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T14:22:30.462 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T14:22:30.479 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-09-13T14:22:30.479 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-13T14:22:30.479 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:22:30.480 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-13T14:22:30.480 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:22:30.480 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:22:30.480 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:22:30.480 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:22:30.788 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T14:22:30.789 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:22:30.789 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T14:22:31.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-13T14:22:31.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:22:31.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:22:31.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T14:22:31.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:22:31.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-13T14:22:31.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:22:31.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:22:31.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T14:22:31.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:22:31.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T14:22:31.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:22:31.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:22:31.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:22:31.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:22:31.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:22:31.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:22:31.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-13T14:22:31.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:22:31.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:22:31.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T14:22:31.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:22:31.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:22:31.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-13T14:22:31.885 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:22:31.885 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:22:32.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-13T14:22:32.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:22:32.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:22:32.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T14:22:32.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:22:32.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:22:32.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:22:32.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:22:32.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:22:32.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:22:32.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:22:32.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:22:32.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:22:32.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:22:32.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:22:32.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:22:32.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:22:32.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:22:32.184 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:22:32.184 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:22:32.184 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:22:32.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:22:32.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:22:32.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:22:32.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:22:32.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:22:32.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:22:32.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:22:32.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:22:32.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:22:32.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:22:32.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-13T14:22:32.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:22:32.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T14:22:32.187 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T14:22:32.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-13T14:22:32.187 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T14:22:32.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T14:22:32.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:22:32.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:22:32.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:22:32.223 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:32.227+0000 7f02a28f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:22:32.247 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:32.251+0000 7f02a28f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:22:32.249 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:32.251+0000 7f02a28f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:22:32.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T14:22:32.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:22:32.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:22:32.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:22:32.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:22:32.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:22:32.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:22:32.514 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:22:32.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:22:32.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:22:32.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:22:33.553 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:33.555+0000 7f02a28f27c0 -1 Falling back to public interface 2024-09-13T14:22:33.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:22:33.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:22:33.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:22:33.878 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:22:33.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:22:33.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:22:34.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:22:35.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:22:35.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:22:35.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:22:35.203 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:22:35.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:22:35.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:22:35.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:22:36.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:22:36.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:22:36.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:22:36.548 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:22:36.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:22:36.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:22:36.639 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:36.643+0000 7f02a28f27c0 -1 osd.0 26 log_to_monitors true 2024-09-13T14:22:36.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:22:37.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:22:37.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:22:37.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:22:37.886 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:22:37.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:22:37.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:22:38.220 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/1448024599,v1:127.0.0.1:6803/1448024599] [v2:127.0.0.1:6804/1448024599,v1:127.0.0.1:6805/1448024599] exists,up 2fc01b1d-60b5-4462-a0b4-3dcf78231e87 2024-09-13T14:22:38.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:22:38.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:22:38.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:22:38.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:22:38.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:22:38.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:22:38.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:22:38.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:22:38.222 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:22:38.222 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:22:38.222 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:22:38.222 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:22:38.222 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:22:38.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:22:38.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:22:38.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:22:38.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:22:38.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:22:38.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:22:38.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:22:38.661 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:22:38.661 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:22:38.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:22:38.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:22:38.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:22:38.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018882 2024-09-13T14:22:38.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018882 2024-09-13T14:22:38.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882' 2024-09-13T14:22:38.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:22:38.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:22:38.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-13T14:22:38.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-13T14:22:38.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672979' 2024-09-13T14:22:38.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:22:38.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:22:39.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-13T14:22:39.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-13T14:22:39.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672979 2-64424509455' 2024-09-13T14:22:39.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:22:39.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:22:39.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018882 2024-09-13T14:22:39.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:22:39.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018882 2024-09-13T14:22:39.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:22:39.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018882 2024-09-13T14:22:39.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-09-13T14:22:39.127 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 128849018882 2024-09-13T14:22:39.127 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:22:39.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 128849018882 2024-09-13T14:22:39.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:22:40.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:22:40.438 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:22:40.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-13T14:22:40.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:22:40.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-13T14:22:40.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:22:40.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:22:40.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-13T14:22:40.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:22:40.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-13T14:22:40.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-13T14:22:40.766 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672979 2024-09-13T14:22:40.766 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:22:41.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-13T14:22:41.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:22:41.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:22:41.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-13T14:22:41.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:22:41.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-13T14:22:41.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:22:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-13T14:22:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-13T14:22:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509455 2024-09-13T14:22:41.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:22:41.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-13T14:22:41.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:22:41.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:22:41.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:22:41.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:22:41.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:22:41.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:22:41.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:22:41.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:22:41.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:22:41.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:22:41.840 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:22:42.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:22:42.172 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:22:42.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:22:42.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:22:42.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:22:42.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:22:42.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:22:42.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-09-13T14:22:42.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-13T14:22:42.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:22:42.571 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-13T14:22:42.571 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T14:22:42.571 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:22:42.571 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:22:42.571 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:22:42.891 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T14:22:42.892 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-13T14:22:42.892 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T14:22:43.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-13T14:22:43.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:22:43.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:22:43.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T14:22:43.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:22:43.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-13T14:22:43.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:22:43.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:22:43.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T14:22:43.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:22:43.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T14:22:43.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:22:43.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:22:43.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:22:43.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:22:43.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:22:43.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:22:43.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-13T14:22:43.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:22:43.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:22:43.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T14:22:43.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:22:43.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:22:43.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-13T14:22:44.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-13T14:22:44.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:22:44.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:22:44.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T14:22:44.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:22:44.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:22:44.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:22:44.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:22:44.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:22:44.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:22:44.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:22:44.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:22:44.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:22:44.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:22:44.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:22:44.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:22:44.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:22:44.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:22:44.452 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:22:44.452 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:22:44.452 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:22:44.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:22:44.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:22:44.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:22:44.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:22:44.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:22:44.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:22:44.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:22:44.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:22:44.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:22:44.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:22:44.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-13T14:22:44.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:22:44.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T14:22:44.454 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T14:22:44.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-13T14:22:44.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T14:22:44.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T14:22:44.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:22:44.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:22:44.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:22:44.491 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:44.495+0000 7f0fb97537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:22:44.494 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:44.495+0000 7f0fb97537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:22:44.495 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:44.499+0000 7f0fb97537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:22:44.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T14:22:44.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:22:44.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:22:44.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:22:44.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:22:44.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:22:44.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:22:44.783 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:22:44.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:22:44.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:22:45.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:22:45.801 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:45.803+0000 7f0fb97537c0 -1 Falling back to public interface 2024-09-13T14:22:46.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:22:46.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:22:46.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:22:46.263 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:22:46.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:22:46.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:22:46.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:22:47.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:22:47.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:22:47.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:22:47.584 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:22:47.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:22:47.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:22:47.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:22:48.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:22:48.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:22:48.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:22:48.911 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:22:48.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:22:48.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:22:48.920 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:48.922+0000 7f0fb97537c0 -1 osd.0 31 log_to_monitors true 2024-09-13T14:22:49.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:22:50.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:22:50.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:22:50.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:22:50.315 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:22:50.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:22:50.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:22:50.649 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/3854243484,v1:127.0.0.1:6803/3854243484] [v2:127.0.0.1:6804/3854243484,v1:127.0.0.1:6805/3854243484] exists,up 2fc01b1d-60b5-4462-a0b4-3dcf78231e87 2024-09-13T14:22:50.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:22:50.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:22:50.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:22:50.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:22:50.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:22:50.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:22:50.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:22:50.651 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:22:50.651 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:22:50.652 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:22:50.652 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:22:50.652 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:22:50.652 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:22:50.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:22:50.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:22:50.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:22:50.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:22:50.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:22:50.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:22:51.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:22:51.087 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:22:51.087 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:22:51.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:22:51.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:22:51.088 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:22:51.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-13T14:22:51.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-13T14:22:51.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362' 2024-09-13T14:22:51.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:22:51.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:22:51.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672982 2024-09-13T14:22:51.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672982 2024-09-13T14:22:51.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362 1-42949672982' 2024-09-13T14:22:51.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:22:51.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:22:51.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-13T14:22:51.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-13T14:22:51.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362 1-42949672982 2-64424509459' 2024-09-13T14:22:51.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:22:51.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855362 2024-09-13T14:22:51.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:22:51.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:22:51.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855362 2024-09-13T14:22:51.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:22:51.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-13T14:22:51.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-09-13T14:22:51.533 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 150323855362 2024-09-13T14:22:51.533 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:22:51.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-13T14:22:51.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:22:51.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672982 2024-09-13T14:22:51.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:22:51.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:22:51.856 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672982 2024-09-13T14:22:51.857 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:22:51.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672982 2024-09-13T14:22:51.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-13T14:22:51.859 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672982 2024-09-13T14:22:51.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:22:52.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-13T14:22:52.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:22:52.210 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-13T14:22:52.211 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:22:52.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:22:52.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-13T14:22:52.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:22:52.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-13T14:22:52.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-13T14:22:52.214 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509459 2024-09-13T14:22:52.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:22:52.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-13T14:22:52.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:22:52.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:22:52.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:22:52.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T14:22:52.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:22:52.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:22:52.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:22:52.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:22:52.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:22:52.940 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:22:52.940 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:22:53.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:22:53.289 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:22:53.289 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:22:53.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:22:53.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T14:22:53.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:22:53.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:22:53.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:22:53.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-13T14:22:53.677 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:22:25.879+0000 7f7c3c564640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-13T14:22:53.678 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:22:25.879+0000 7f7c3c564640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-13T14:22:53.678 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:22:25.879+0000 7f7c3c564640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-13T14:22:53.678 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:22:25.879+0000 7f7c3c564640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-09-13T14:22:53.678 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T14:22:25.879+0000 7f7c3c564640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-13T14:22:53.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T14:22:53.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:22:53.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:22:53.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:22:53.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:22:53.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:22:53.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:22:53.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:22:53.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:22:53.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:22:53.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:22:53.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:22:53.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:22:53.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:22:53.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:22:53.993 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:22:53.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:22:53.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:22:53.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:22:53.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:22:53.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:22:53.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:22:53.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:22:54.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:22:54.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:22:54.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:22:54.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:22:54.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:22:54.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:22:54.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T14:22:54.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T14:22:54.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T14:22:54.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T14:22:54.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:22:54.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:22:54.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:22:54.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:22:54.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:22:54.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:22:54.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:22:54.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:22:54.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:22:54.016 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:22:54.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:22:54.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:22:54.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:22:54.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:22:54.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:22:54.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:22:54.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:22:54.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:22:54.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:22:54.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:22:54.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:22:54.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:22:54.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:22:54.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:22:54.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:22:54.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:22:54.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:22:54.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:22:54.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T14:22:54.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:22:54.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:22:54.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:22:54.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T14:22:54.030 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:22:54.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T14:22:54.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:22:54.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T14:22:54.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-09-13T14:22:54.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-09-13T14:22:54.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-13T14:22:54.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-09-13T14:22:54.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-09-13T14:22:54.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-13T14:22:54.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T14:22:54.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T14:22:54.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T14:22:54.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T14:22:54.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T14:22:54.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T14:22:54.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T14:22:54.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:22:54.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:22:54.528 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:22:54.528 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:22:54.528 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:22:54.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:22:54.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T14:22:54.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T14:22:54.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T14:22:54.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:22:54.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:22:54.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T14:22:54.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T14:22:54.749 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:22:54.749 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:22:54.749 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:22:54.750 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:22:54.750 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:22:54.750 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:22:54.750 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:22:54.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:22:54.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T14:22:54.866 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T14:22:54.866 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:22:54.866 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:22:54.866 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T14:22:54.867 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T14:22:54.867 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:22:54.867 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:22:54.867 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:22:54.867 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:22:54.868 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:22:54.868 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:22:54.868 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:22:54.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:22:54.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T14:22:54.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-13T14:22:54.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T14:22:54.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T14:22:54.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T14:22:54.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T14:22:54.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T14:22:54.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T14:22:55.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T14:22:55.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:22:55.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:22:55.839 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:22:55.839 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:22:55.839 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:22:55.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:22:55.840 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:22:55.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:22:55.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:22:55.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-09-13T14:22:55.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-13T14:22:55.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:22:55.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:22:55.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:22:55.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T14:22:55.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:22:55.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:22:55.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:22:55.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:22:55.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:22:55.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:22:55.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:22:55.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:22:55.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:22:55.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:22:55.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:22:55.879 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:22:55.879 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:22:55.879 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:22:55.879 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:22:55.880 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:22:55.880 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:22:55.880 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:22:55.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:22:55.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:22:55.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:22:55.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:22:55.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:22:55.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:22:55.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:22:55.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:22:55.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:22:55.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:22:55.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:22:55.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:22:55.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:22:55.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:22:55.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=75a835d1-fc96-4f6b-b56b-7459d19a350a 2024-09-13T14:22:55.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 75a835d1-fc96-4f6b-b56b-7459d19a350a' 2024-09-13T14:22:55.882 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 75a835d1-fc96-4f6b-b56b-7459d19a350a 2024-09-13T14:22:55.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:22:55.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC/SuRm0KrENRAA5iG2SnRRzpsgmbMAgL9BGw== 2024-09-13T14:22:55.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC/SuRm0KrENRAA5iG2SnRRzpsgmbMAgL9BGw=="}' 2024-09-13T14:22:55.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 75a835d1-fc96-4f6b-b56b-7459d19a350a -i td/osd-scrub-repair/0/new.json 2024-09-13T14:22:56.166 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:22:56.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T14:22:56.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQC/SuRm0KrENRAA5iG2SnRRzpsgmbMAgL9BGw== --osd-uuid 75a835d1-fc96-4f6b-b56b-7459d19a350a 2024-09-13T14:22:56.220 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:56.222+0000 7f56dd0897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:22:56.234 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:56.234+0000 7f56dd0897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:22:56.235 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:56.238+0000 7f56dd0897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:22:56.236 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:56.238+0000 7f56dd0897c0 -1 bdev(0x55a629dedc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:22:56.236 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:22:56.238+0000 7f56dd0897c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T14:23:02.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T14:23:02.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:23:02.461 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:23:02.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T14:23:02.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:23:02.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T14:23:02.854 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:23:02.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:23:02.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:23:02.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:23:02.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:23:02.890 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:23:02.890+0000 7f67a84757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:23:02.892 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:23:02.894+0000 7f67a84757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:23:02.894 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:23:02.894+0000 7f67a84757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:23:03.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T14:23:03.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:23:03.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:23:03.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:23:03.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:23:03.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:03.162 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:23:03.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:23:03.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:03.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:03.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:03.957 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:23:03.958+0000 7f67a84757c0 -1 Falling back to public interface 2024-09-13T14:23:04.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:04.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:04.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:23:04.518 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:23:04.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:04.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:04.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:05.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:05.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:05.830 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:23:05.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:23:05.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:05.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:06.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:06.760 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:23:06.762+0000 7f67a84757c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:23:07.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:07.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:07.152 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:23:07.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:23:07.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:07.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:07.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:08.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:08.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:08.488 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:23:08.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:23:08.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:08.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:08.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:09.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:09.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:09.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:23:09.842 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:23:09.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:09.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:10.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:11.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:11.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:11.179 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:23:11.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:23:11.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:11.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:11.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:12.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:12.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:12.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:23:12.618 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:23:12.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:12.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:12.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:13.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:13.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:13.952 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:23:13.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:23:13.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:13.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:14.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:15.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:15.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:15.290 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:23:15.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:23:15.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:15.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:15.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:16.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:16.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:16.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:23:16.786 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:23:16.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:16.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:17.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:18.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:18.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:18.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:23:18.125 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:23:18.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:18.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:18.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:19.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:19.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:19.453 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:23:19.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:23:19.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:19.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:20.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:21.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:21.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:21.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:23:21.227 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:23:21.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:21.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:22.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:23.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:23.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:23.057 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:23:23.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:23:23.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:23.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:24.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:25.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:25.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:25.142 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:23:25.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:23:25.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:25.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:25.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:26.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:26.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:26.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:23:26.635 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:23:26.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:26.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:27.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:28.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:28.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:28.170 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:23:28.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:23:28.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:28.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:28.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:29.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:29.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:29.744 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:23:29.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:23:29.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:29.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:30.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:31.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:31.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:31.202 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:23:31.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:23:31.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:31.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:23:31.539 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/939124398,v1:127.0.0.1:6803/939124398] [v2:127.0.0.1:6804/939124398,v1:127.0.0.1:6805/939124398] exists,up 75a835d1-fc96-4f6b-b56b-7459d19a350a 2024-09-13T14:23:31.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:23:31.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:23:31.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:23:31.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-13T14:23:31.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:23:31.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:23:31.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:23:31.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T14:23:31.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:23:31.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:23:31.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:23:31.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:23:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:23:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:23:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:23:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:23:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:23:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:23:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:23:31.543 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:23:31.543 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:23:31.543 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:23:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:23:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:23:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:23:31.544 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:23:31.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:23:31.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:23:31.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:23:31.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:23:31.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:23:31.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:23:31.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:23:31.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:23:31.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:23:31.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:23:31.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:23:31.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:23:31.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:23:31.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:23:31.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=aeb9de6f-4b5d-4df7-9688-99097c5e26e6 2024-09-13T14:23:31.545 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 aeb9de6f-4b5d-4df7-9688-99097c5e26e6 2024-09-13T14:23:31.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 aeb9de6f-4b5d-4df7-9688-99097c5e26e6' 2024-09-13T14:23:31.546 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:23:31.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDjSuRmPoz5IRAAtD0oI0XV4YZs8aokJ5SDnA== 2024-09-13T14:23:31.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDjSuRmPoz5IRAAtD0oI0XV4YZs8aokJ5SDnA=="}' 2024-09-13T14:23:31.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new aeb9de6f-4b5d-4df7-9688-99097c5e26e6 -i td/osd-scrub-repair/1/new.json 2024-09-13T14:23:31.948 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:23:31.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T14:23:31.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDjSuRmPoz5IRAAtD0oI0XV4YZs8aokJ5SDnA== --osd-uuid aeb9de6f-4b5d-4df7-9688-99097c5e26e6 2024-09-13T14:23:32.003 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:23:32.005+0000 7f0a88e1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:23:32.050 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:23:32.049+0000 7f0a88e1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:23:32.052 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:23:32.053+0000 7f0a88e1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:23:32.053 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:23:32.053+0000 7f0a88e1b7c0 -1 bdev(0x55d757679c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:23:32.053 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:23:32.053+0000 7f0a88e1b7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T14:23:38.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T14:23:38.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:23:38.690 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:23:38.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T14:23:38.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:23:39.100 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:23:39.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T14:23:39.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:23:39.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:23:39.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:23:39.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:23:39.138 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:23:39.137+0000 7f1ebd6047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:23:39.141 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:23:39.141+0000 7f1ebd6047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:23:39.141 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:23:39.141+0000 7f1ebd6047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:23:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T14:23:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:23:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:23:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:23:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:23:39.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:39.412 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:23:39.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:23:39.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:39.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:23:39.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:40.198 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:23:40.197+0000 7f1ebd6047c0 -1 Falling back to public interface 2024-09-13T14:23:40.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:40.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:40.756 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:23:40.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:23:40.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:40.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:23:41.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:42.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:42.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:42.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:23:42.078 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:23:42.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:42.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:23:42.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:43.160 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:23:43.161+0000 7f1ebd6047c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:23:43.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:43.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:43.392 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:23:43.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:23:43.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:43.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:23:43.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:44.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:44.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:44.894 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:23:44.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:23:44.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:44.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:23:45.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:46.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:46.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:46.375 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:23:46.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:23:46.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:46.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:23:46.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:47.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:47.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:47.718 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:23:47.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:23:47.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:47.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:23:48.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:49.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:49.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:49.220 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:23:49.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:23:49.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:49.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:23:49.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:50.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:50.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:50.593 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:23:50.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:23:50.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:50.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:23:51.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:52.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:52.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:52.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:23:52.073 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:23:52.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:52.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:23:52.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:53.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:53.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:53.408 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:23:53.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:23:53.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:53.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:23:53.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:54.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:54.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:54.827 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:23:54.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:23:54.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:54.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:23:55.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:56.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:56.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:56.385 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:23:56.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:23:56.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:56.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:23:56.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:57.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:57.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:57.724 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:23:57.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:23:57.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:57.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:23:58.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:23:59.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:23:59.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:23:59.806 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:23:59.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:23:59.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:23:59.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:24:00.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:01.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:01.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:01.499 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:24:01.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:24:01.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:01.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:24:02.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:03.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:03.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:03.289 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:24:03.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:24:03.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:03.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:24:03.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:04.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:04.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:04.652 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:24:04.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:24:04.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:04.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:24:05.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:06.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:06.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:06.332 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:24:06.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:24:06.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:06.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:24:07.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:08.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:08.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:08.279 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:24:08.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:24:08.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:08.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:24:08.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:09.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:09.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:09.752 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T14:24:09.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T14:24:09.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:09.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:24:10.088 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3545748664,v1:127.0.0.1:6811/3545748664] [v2:127.0.0.1:6812/3545748664,v1:127.0.0.1:6813/3545748664] exists,up aeb9de6f-4b5d-4df7-9688-99097c5e26e6 2024-09-13T14:24:10.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:24:10.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:24:10.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:24:10.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-13T14:24:10.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:24:10.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:24:10.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:24:10.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T14:24:10.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:24:10.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:24:10.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:24:10.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:24:10.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:24:10.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:24:10.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T14:24:10.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T14:24:10.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:24:10.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:24:10.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:24:10.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:24:10.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:24:10.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:24:10.095 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:24:10.095 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:24:10.095 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:24:10.095 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:24:10.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:24:10.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:24:10.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:24:10.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:24:10.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:24:10.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:24:10.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:24:10.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:24:10.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:24:10.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:24:10.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:24:10.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:24:10.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T14:24:10.096 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:24:10.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6f2c4bf4-f557-48ef-8a10-632f3bfadd84 2024-09-13T14:24:10.097 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 6f2c4bf4-f557-48ef-8a10-632f3bfadd84 2024-09-13T14:24:10.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6f2c4bf4-f557-48ef-8a10-632f3bfadd84' 2024-09-13T14:24:10.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:24:10.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAKS+RmhKDwBhAAMwp6R8NxmwoeZpaeeGTrtA== 2024-09-13T14:24:10.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAKS+RmhKDwBhAAMwp6R8NxmwoeZpaeeGTrtA=="}' 2024-09-13T14:24:10.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6f2c4bf4-f557-48ef-8a10-632f3bfadd84 -i td/osd-scrub-repair/2/new.json 2024-09-13T14:24:10.464 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:24:10.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T14:24:10.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAKS+RmhKDwBhAAMwp6R8NxmwoeZpaeeGTrtA== --osd-uuid 6f2c4bf4-f557-48ef-8a10-632f3bfadd84 2024-09-13T14:24:10.521 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:24:10.524+0000 7ff38e44f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:24:10.553 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:24:10.556+0000 7ff38e44f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:24:10.555 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:24:10.560+0000 7ff38e44f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:24:10.555 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:24:10.560+0000 7ff38e44f7c0 -1 bdev(0x5567081e5c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T14:24:10.555 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:24:10.560+0000 7ff38e44f7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T14:24:17.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T14:24:17.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:24:17.019 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T14:24:17.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T14:24:17.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:24:17.444 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T14:24:17.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T14:24:17.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:24:17.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:24:17.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:24:17.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:24:17.481 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:24:17.484+0000 7f8b0198b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:24:17.509 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:24:17.512+0000 7f8b0198b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:24:17.511 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:24:17.516+0000 7f8b0198b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:24:17.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T14:24:17.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:24:17.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:24:17.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:24:17.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:24:17.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:17.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:24:17.828 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:24:17.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:17.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:18.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:18.586 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:24:18.588+0000 7f8b0198b7c0 -1 Falling back to public interface 2024-09-13T14:24:19.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:19.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:19.171 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:24:19.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:24:19.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:19.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:19.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:20.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:20.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:20.517 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:24:20.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:24:20.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:20.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:20.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:21.494 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:24:21.496+0000 7f8b0198b7c0 -1 osd.2 0 log_to_monitors true 2024-09-13T14:24:21.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:21.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:21.844 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:24:21.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:24:21.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:21.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:22.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:23.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:23.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:23.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:24:23.240 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:24:23.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:23.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:23.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:24.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:24.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:24.577 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:24:24.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:24:24.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:24.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:24.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:25.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:25.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:25.940 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:24:25.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:24:25.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:25.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:26.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:27.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:27.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:27.393 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:24:27.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:24:27.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:27.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:27.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:28.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:28.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:28.899 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:24:28.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:24:28.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:28.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:29.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:30.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:30.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:30.220 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:24:30.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:24:30.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:30.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:30.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:31.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:31.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:31.671 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:24:31.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:24:31.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:31.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:32.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:33.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:33.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:33.009 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:24:33.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:24:33.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:33.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:33.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:34.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:34.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:34.535 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:24:34.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:24:34.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:34.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:34.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:35.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:35.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:35.889 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:24:35.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:24:35.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:35.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:36.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:37.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:37.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:37.638 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:24:37.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:24:37.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:37.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:38.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:39.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:39.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:39.061 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:24:39.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:24:39.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:39.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:39.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:40.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:40.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:40.998 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:24:40.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:24:40.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:40.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:41.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:42.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:42.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:42.933 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:24:42.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:24:42.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:42.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:43.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:44.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:44.255 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:24:44.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:44.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:24:44.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:44.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:45.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:46.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:46.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:46.313 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:24:46.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:24:46.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:46.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:46.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:24:47.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:24:47.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:24:47.874 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T14:24:47.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T14:24:47.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:24:47.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:24:48.247 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1952782682,v1:127.0.0.1:6819/1952782682] [v2:127.0.0.1:6820/1952782682,v1:127.0.0.1:6821/1952782682] exists,up 6f2c4bf4-f557-48ef-8a10-632f3bfadd84 2024-09-13T14:24:48.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:24:48.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:24:48.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:24:48.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-09-13T14:24:48.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T14:24:48.562 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' does not exist 2024-09-13T14:24:48.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T14:24:48.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T14:24:49.007 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T14:24:49.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:24:50.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T14:24:52.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-09-13T14:24:52.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:24:52.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:24:52.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:24:52.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:24:52.377 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:24:52.377 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:24:52.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:24:52.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:24:52.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:24:52.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:24:52.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:24:52.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:24:52.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:24:52.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:24:52.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:24:52.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:24:52.885 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:24:52.885 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:24:52.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:24:52.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:24:52.886 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:24:53.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-13T14:24:53.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-13T14:24:53.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-13T14:24:53.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:24:53.062 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:24:53.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=73014444044 2024-09-13T14:24:53.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 73014444044 2024-09-13T14:24:53.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-73014444044' 2024-09-13T14:24:53.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:24:53.198 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:24:53.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822663 2024-09-13T14:24:53.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822663 2024-09-13T14:24:53.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-73014444044 2-154618822663' 2024-09-13T14:24:53.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:24:53.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-13T14:24:53.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:24:53.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:24:53.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-13T14:24:53.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:24:53.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-13T14:24:53.333 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836507 2024-09-13T14:24:53.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-13T14:24:53.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:24:53.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836507 2024-09-13T14:24:53.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:24:54.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:24:54.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:24:54.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-13T14:24:54.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:24:54.961 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:24:54.961 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-73014444044 2024-09-13T14:24:54.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:24:54.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-73014444044 2024-09-13T14:24:54.963 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:24:54.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=73014444044 2024-09-13T14:24:54.964 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 73014444044 2024-09-13T14:24:54.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 73014444044' 2024-09-13T14:24:54.964 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:24:55.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 73014444044 -lt 73014444044 2024-09-13T14:24:55.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:24:55.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-154618822663 2024-09-13T14:24:55.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:24:55.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:24:55.280 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:24:55.280 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-154618822663 2024-09-13T14:24:55.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822663 2024-09-13T14:24:55.282 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 154618822663 2024-09-13T14:24:55.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 154618822663' 2024-09-13T14:24:55.282 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:24:55.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822664 -lt 154618822663 2024-09-13T14:24:55.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:24:55.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:24:55.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:24:56.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T14:24:56.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:24:56.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:24:56.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:24:56.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:24:56.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:24:56.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:24:56.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:24:56.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:24:56.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:24:56.419 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:24:56.419 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:24:56.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T14:24:56.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:24:56.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:24:56.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-09-13T14:24:56.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-13T14:24:56.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-13T14:24:56.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-13T14:24:56.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-13T14:24:56.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-13T14:24:57.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-13T14:24:57.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-13T14:24:58.020 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'ecpool' created 2024-09-13T14:24:58.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:24:59.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-13T14:24:59.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-13T14:24:59.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:24:59.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:24:59.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:24:59.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:24:59.046 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:24:59.046 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:24:59.046 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:24:59.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:24:59.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:24:59.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:24:59.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:24:59.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:24:59.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:24:59.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:24:59.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:24:59.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:24:59.507 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:24:59.507 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:24:59.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:24:59.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:24:59.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:24:59.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836509 2024-09-13T14:24:59.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836509 2024-09-13T14:24:59.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509' 2024-09-13T14:24:59.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:24:59.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:24:59.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=73014444047 2024-09-13T14:24:59.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 73014444047 2024-09-13T14:24:59.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-73014444047' 2024-09-13T14:24:59.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:24:59.828 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:24:59.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822666 2024-09-13T14:24:59.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822666 2024-09-13T14:24:59.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-73014444047 2-154618822666' 2024-09-13T14:24:59.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:24:59.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836509 2024-09-13T14:24:59.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:24:59.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:24:59.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836509 2024-09-13T14:24:59.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:24:59.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836509 2024-09-13T14:24:59.971 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836509 2024-09-13T14:24:59.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-13T14:24:59.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:25:00.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836509 2024-09-13T14:25:00.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:25:01.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:25:01.280 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:25:01.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-13T14:25:01.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:25:01.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-73014444047 2024-09-13T14:25:01.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:25:01.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:25:01.603 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-73014444047 2024-09-13T14:25:01.603 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:25:01.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=73014444047 2024-09-13T14:25:01.604 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 73014444047 2024-09-13T14:25:01.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 73014444047' 2024-09-13T14:25:01.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:25:01.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 73014444047 -lt 73014444047 2024-09-13T14:25:01.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:25:01.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:25:01.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-154618822666 2024-09-13T14:25:01.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:25:01.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-154618822666 2024-09-13T14:25:01.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:25:01.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822666 2024-09-13T14:25:01.985 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 154618822666 2024-09-13T14:25:01.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 154618822666' 2024-09-13T14:25:01.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:25:02.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822666 -lt 154618822666 2024-09-13T14:25:02.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:25:02.370 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:25:02.370 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:25:02.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:25:02.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:25:02.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:25:02.751 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:25:02.751 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:25:02.751 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:25:02.751 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:25:02.751 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:25:03.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:25:03.062 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:25:03.062 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:25:03.062 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:25:03.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:25:03.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:25:03.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:25:03.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-13T14:25:03.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-09-13T14:25:03.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-09-13T14:25:03.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-13T14:25:03.789 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-13T14:25:03.789 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-13T14:25:03.789 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:25:03.790 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-13T14:25:03.790 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-13T14:25:03.790 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:25:03.791 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:25:03.791 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:25:04.151 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-13T14:25:04.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:25:04.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-13T14:25:04.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:25:04.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:25:04.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:25:04.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:25:04.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:25:04.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:25:04.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:25:04.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:25:04.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:25:04.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:25:04.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:25:04.470 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:25:04.471 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:25:04.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:25:04.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:25:04.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:25:04.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:25:04.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:25:04.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:25:04.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:25:04.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:25:04.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:25:04.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:25:04.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-13T14:25:07.066 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-13T14:25:07.066 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-13T14:25:07.067 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-13T14:25:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:25:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:25:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:25:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:25:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:25:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:25:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:25:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:25:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:25:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:25:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:25:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:25:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:25:07.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:25:07.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:25:07.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:25:07.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:25:07.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:25:07.608 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:25:07.608 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:25:07.608 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:25:07.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:25:07.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:25:07.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:25:07.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:25:07.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:25:07.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:25:07.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:25:07.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:25:07.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:25:07.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:25:07.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:25:07.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:25:07.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:25:07.609 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T14:25:07.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:25:07.609 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:25:07.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:25:07.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:25:07.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:25:07.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:25:07.647 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:07.650+0000 7fc31a6337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:25:07.650 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:07.650+0000 7fc31a6337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:25:07.652 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:07.654+0000 7fc31a6337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:25:07.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:25:07.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:25:07.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:25:07.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:25:07.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:25:07.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:07.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:25:07.942 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:25:07.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:07.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:25:08.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:09.170 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:09.170+0000 7fc31a6337c0 -1 Falling back to public interface 2024-09-13T14:25:09.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:09.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:09.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:25:09.302 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:25:09.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:09.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:25:09.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:10.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:10.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:10.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:25:10.727 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:25:10.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:10.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:25:11.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:12.040 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:12.042+0000 7fc31a6337c0 -1 osd.1 56 log_to_monitors true 2024-09-13T14:25:12.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:12.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:12.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:25:12.071 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:25:12.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:12.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:25:12.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:13.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:13.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:13.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:25:13.451 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:25:13.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:13.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:25:13.788 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 63 up_thru 63 down_at 57 last_clean_interval [17,56) [v2:127.0.0.1:6810/2611672694,v1:127.0.0.1:6811/2611672694] [v2:127.0.0.1:6812/2611672694,v1:127.0.0.1:6813/2611672694] exists,up aeb9de6f-4b5d-4df7-9688-99097c5e26e6 2024-09-13T14:25:13.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:25:13.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:25:13.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:25:13.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:25:13.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:25:13.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:25:13.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:25:13.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:25:13.790 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:25:13.790 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:25:13.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:25:13.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:25:13.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:25:13.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:25:13.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:25:13.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:25:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:25:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:25:13.894 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:25:14.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:25:14.269 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:25:14.269 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:25:14.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:25:14.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:25:14.270 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:25:14.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836513 2024-09-13T14:25:14.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836513 2024-09-13T14:25:14.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513' 2024-09-13T14:25:14.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:25:14.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:25:14.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=270582939650 2024-09-13T14:25:14.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 270582939650 2024-09-13T14:25:14.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-270582939650' 2024-09-13T14:25:14.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:25:14.562 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:25:14.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822670 2024-09-13T14:25:14.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822670 2024-09-13T14:25:14.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-270582939650 2-154618822670' 2024-09-13T14:25:14.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:25:14.704 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836513 2024-09-13T14:25:14.704 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:25:14.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:25:14.706 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836513 2024-09-13T14:25:14.706 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:25:14.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836513 2024-09-13T14:25:14.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-13T14:25:14.708 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836513 2024-09-13T14:25:14.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:25:15.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836513 -lt 21474836513 2024-09-13T14:25:15.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:25:15.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-270582939650 2024-09-13T14:25:15.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:25:15.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:25:15.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-270582939650 2024-09-13T14:25:15.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:25:15.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=270582939650 2024-09-13T14:25:15.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 270582939650' 2024-09-13T14:25:15.022 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 270582939650 2024-09-13T14:25:15.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:25:15.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939650 -lt 270582939650 2024-09-13T14:25:15.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:25:15.364 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:25:15.365 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-154618822670 2024-09-13T14:25:15.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:25:15.367 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-154618822670 2024-09-13T14:25:15.367 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:25:15.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822670 2024-09-13T14:25:15.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 154618822670' 2024-09-13T14:25:15.369 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 154618822670 2024-09-13T14:25:15.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:25:15.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822669 -lt 154618822670 2024-09-13T14:25:15.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:25:16.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:25:16.785 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:25:17.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822670 -lt 154618822670 2024-09-13T14:25:17.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:25:17.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:25:17.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:25:17.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:25:17.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:25:17.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:25:17.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:25:17.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:25:17.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:25:17.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:25:17.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:25:17.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:25:17.899 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:25:17.899 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:25:17.899 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:25:18.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:25:18.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:25:18.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:25:18.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-09-13T14:25:18.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:25:18.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:25:18.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:25:18.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:25:18.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-09-13T14:25:18.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-09-13T14:25:18.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:25:18.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:25:18.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:25:18.689 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:25:19.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-09-13T14:24:57.885970+0000 2024-09-13T14:25:19.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:25:19.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:24:57.885970+0000 2024-09-13T14:25:19.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:25:19.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:25:19.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:25:19.012 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:25:19.012 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:25:19.012 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:25:19.012 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:25:19.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:25:19.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:24:57.885970+0000 '>' 2024-09-13T14:24:57.885970+0000 2024-09-13T14:25:19.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:25:20.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:25:20.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:25:20.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:25:20.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:25:20.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:25:20.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:25:20.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:25:20.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:25:17.864313+0000 '>' 2024-09-13T14:24:57.885970+0000 2024-09-13T14:25:20.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:25:20.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-09-13T14:25:20.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:25:20.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:25:20.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:25:20.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:25:20.661 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:25:20.661 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:25:20.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:25:20.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:25:20.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:25:20.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:25:20.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:25:20.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:25:20.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:25:20.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:25:20.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:25:21.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:25:21.132 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:25:21.133 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:25:21.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:25:21.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:25:21.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:25:21.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836516 2024-09-13T14:25:21.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836516 2024-09-13T14:25:21.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516' 2024-09-13T14:25:21.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:25:21.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:25:21.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=270582939652 2024-09-13T14:25:21.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 270582939652 2024-09-13T14:25:21.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516 1-270582939652' 2024-09-13T14:25:21.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:25:21.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:25:21.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822672 2024-09-13T14:25:21.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822672 2024-09-13T14:25:21.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516 1-270582939652 2-154618822672' 2024-09-13T14:25:21.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:25:21.563 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836516 2024-09-13T14:25:21.563 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:25:21.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:25:21.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836516 2024-09-13T14:25:21.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:25:21.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836516 2024-09-13T14:25:21.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-09-13T14:25:21.566 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836516 2024-09-13T14:25:21.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:25:21.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836514 -lt 21474836516 2024-09-13T14:25:21.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:25:22.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:25:22.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:25:23.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836516 -lt 21474836516 2024-09-13T14:25:23.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:25:23.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-270582939652 2024-09-13T14:25:23.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:25:23.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:25:23.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-270582939652 2024-09-13T14:25:23.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:25:23.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=270582939652 2024-09-13T14:25:23.274 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 270582939652 2024-09-13T14:25:23.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 270582939652' 2024-09-13T14:25:23.275 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:25:23.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939653 -lt 270582939652 2024-09-13T14:25:23.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:25:23.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-154618822672 2024-09-13T14:25:23.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:25:23.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:25:23.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-154618822672 2024-09-13T14:25:23.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:25:23.603 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 154618822672 2024-09-13T14:25:23.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822672 2024-09-13T14:25:23.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 154618822672' 2024-09-13T14:25:23.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:25:23.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822672 -lt 154618822672 2024-09-13T14:25:23.976 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:25:23.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:25:23.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:25:24.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:25:24.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:25:24.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:25:24.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:25:24.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:25:24.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:25:24.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:25:24.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:25:24.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:25:24.735 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:25:24.735 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:25:24.735 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:25:25.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:25:25.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:25:25.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:25:25.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-13T14:25:25.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-13T14:25:25.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:25:25.203 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-13T14:25:25.203 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-13T14:25:25.203 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:25:25.204 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:25:25.204 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:25:25.562 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-13T14:25:25.562 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:25:25.562 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-13T14:25:25.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:25:25.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:25:25.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:25:25.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:25:25.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:25:25.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:25:25.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:25:25.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:25:25.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:25:25.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:25:25.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:25:25.900 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:25:25.900 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:25:25.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:25:25.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:25:25.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:25:27.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:25:27.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:25:27.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:25:27.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:25:27.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:25:27.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:25:27.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:25:27.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-13T14:25:27.693 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-13T14:25:27.693 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-13T14:25:27.694 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:25:27.694 INFO:tasks.workunit.client.0.smithi112.stdout:hinfo_key 2024-09-13T14:25:27.694 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:25:27.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:25:27.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:25:27.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:25:27.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:25:27.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:25:27.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:25:27.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:25:27.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:25:27.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:25:27.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:25:27.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:25:27.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:25:27.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:25:27.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:25:27.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:25:27.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:25:27.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:25:27.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:25:27.982 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:25:27.982 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:25:27.982 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:25:27.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:25:27.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:25:27.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:25:27.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:25:27.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:25:27.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:25:27.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:25:27.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:25:27.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:25:27.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:25:27.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:25:27.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:25:27.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:25:27.984 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T14:25:27.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:25:27.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:25:27.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:25:27.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:25:27.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:25:27.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:25:28.021 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:28.021+0000 7fc0787357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:25:28.023 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:28.025+0000 7fc0787357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:25:28.025 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:28.025+0000 7fc0787357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:25:28.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:25:28.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:25:28.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:25:28.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:25:28.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:25:28.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:28.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:25:28.307 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:25:28.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:28.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:25:28.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:29.344 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:29.341+0000 7fc0787357c0 -1 Falling back to public interface 2024-09-13T14:25:29.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:29.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:29.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:25:29.655 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:25:29.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:29.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:25:29.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:30.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:30.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:30.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:25:30.983 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:25:30.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:30.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:25:31.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:32.329 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:32.329+0000 7fc0787357c0 -1 osd.1 72 log_to_monitors true 2024-09-13T14:25:32.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:32.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:32.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:25:32.357 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:25:32.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:32.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:25:32.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:33.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:33.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:33.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:25:33.777 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:25:33.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:33.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:25:34.130 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 78 up_thru 78 down_at 73 last_clean_interval [63,72) [v2:127.0.0.1:6810/3624877354,v1:127.0.0.1:6811/3624877354] [v2:127.0.0.1:6812/3624877354,v1:127.0.0.1:6813/3624877354] exists,up aeb9de6f-4b5d-4df7-9688-99097c5e26e6 2024-09-13T14:25:34.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:25:34.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:25:34.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:25:34.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:25:34.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:25:34.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:25:34.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:25:34.131 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:25:34.132 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:25:34.132 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:25:34.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:25:34.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:25:34.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:25:34.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:25:34.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:25:34.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:25:34.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:25:34.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:25:34.240 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:25:34.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:25:34.672 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:25:34.672 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:25:34.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:25:34.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:25:34.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:25:34.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836519 2024-09-13T14:25:34.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836519 2024-09-13T14:25:34.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519' 2024-09-13T14:25:34.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:25:34.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:25:34.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449090 2024-09-13T14:25:34.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449090 2024-09-13T14:25:34.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519 1-335007449090' 2024-09-13T14:25:34.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:25:34.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:25:35.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822676 2024-09-13T14:25:35.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822676 2024-09-13T14:25:35.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519 1-335007449090 2-154618822676' 2024-09-13T14:25:35.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:25:35.155 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836519 2024-09-13T14:25:35.155 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:25:35.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:25:35.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836519 2024-09-13T14:25:35.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:25:35.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836519 2024-09-13T14:25:35.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-09-13T14:25:35.159 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836519 2024-09-13T14:25:35.159 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:25:35.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836518 -lt 21474836519 2024-09-13T14:25:35.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:25:36.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:25:36.476 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:25:36.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836520 -lt 21474836519 2024-09-13T14:25:36.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:25:36.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-335007449090 2024-09-13T14:25:36.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:25:36.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:25:36.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-335007449090 2024-09-13T14:25:36.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:25:36.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449090 2024-09-13T14:25:36.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 335007449090' 2024-09-13T14:25:36.819 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 335007449090 2024-09-13T14:25:36.820 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:25:37.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449090 -lt 335007449090 2024-09-13T14:25:37.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:25:37.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-154618822676 2024-09-13T14:25:37.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:25:37.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:25:37.151 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-154618822676 2024-09-13T14:25:37.151 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:25:37.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822676 2024-09-13T14:25:37.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 154618822676' 2024-09-13T14:25:37.152 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 154618822676 2024-09-13T14:25:37.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:25:37.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822676 -lt 154618822676 2024-09-13T14:25:37.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:25:37.492 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:25:37.492 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:25:37.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:25:37.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:25:37.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:25:37.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:25:37.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:25:37.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:25:37.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:25:37.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:25:38.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:25:38.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:25:38.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:25:38.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:25:38.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:25:38.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:25:38.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:25:38.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:25:38.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:25:38.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T14:25:38.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:25:38.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:25:38.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:25:38.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:25:38.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:25:38.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:25:38.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:25:38.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:25:38.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:25:38.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:25:38.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:25:38.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:25:38.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:25:38.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:25:38.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:25:38.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:25:38.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:25:38.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:25:38.767 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:25:38.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:25:38.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:25:38.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:25:38.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:25:38.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:25:38.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:25:38.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:25:38.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:25:38.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:25:38.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T14:25:38.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T14:25:38.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T14:25:38.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T14:25:38.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:25:38.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:25:38.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:25:38.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:25:38.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:25:38.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:25:38.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:25:38.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:25:38.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:25:38.791 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:25:38.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:25:38.792 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:25:38.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:25:38.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:25:38.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:25:38.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:25:38.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:25:38.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:25:38.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:25:38.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:25:38.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:25:38.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:25:38.798 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:25:38.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:25:38.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:25:38.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:25:38.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:25:38.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:25:38.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T14:25:38.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:25:38.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:25:38.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:25:38.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T14:25:38.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:25:38.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T14:25:38.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:25:38.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T14:25:38.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-09-13T14:25:38.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-09-13T14:25:38.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-09-13T14:25:38.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-13T14:25:38.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-09-13T14:25:38.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-09-13T14:25:38.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-13T14:25:38.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T14:25:38.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T14:25:38.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T14:25:38.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T14:25:38.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T14:25:38.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T14:25:39.280 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T14:25:39.280 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:25:39.280 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:25:39.282 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:25:39.282 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:25:39.282 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:25:39.282 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:25:39.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T14:25:39.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T14:25:39.492 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T14:25:39.492 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:25:39.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:25:39.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T14:25:39.494 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T14:25:39.494 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:25:39.494 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:25:39.494 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:25:39.494 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:25:39.494 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:25:39.494 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:25:39.494 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:25:39.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:25:39.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T14:25:39.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T14:25:39.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:25:39.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:25:39.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T14:25:39.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T14:25:39.612 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:25:39.612 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:25:39.612 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:25:39.613 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:25:39.613 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:25:39.613 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:25:39.613 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:25:39.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:25:39.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T14:25:39.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-13T14:25:39.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T14:25:39.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T14:25:39.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T14:25:39.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T14:25:39.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T14:25:39.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T14:25:40.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T14:25:40.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:25:40.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:25:40.599 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:25:40.599 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:25:40.599 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:25:40.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:25:40.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:25:40.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:25:40.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:25:40.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-09-13T14:25:40.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-13T14:25:40.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:25:40.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:25:40.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:25:40.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T14:25:40.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:25:40.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:25:40.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:25:40.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:25:40.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:25:40.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:25:40.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:25:40.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:25:40.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:25:40.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:25:40.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:25:40.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:25:40.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:25:40.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:25:40.646 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:25:40.646 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:25:40.646 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:25:40.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:25:40.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:25:40.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:25:40.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:25:40.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:25:40.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:25:40.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:25:40.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:25:40.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:25:40.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:25:40.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:25:40.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:25:40.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:25:40.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:25:40.647 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:25:40.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e580767c-b65f-4126-a98e-f84c273a1fb6 2024-09-13T14:25:40.648 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 e580767c-b65f-4126-a98e-f84c273a1fb6 2024-09-13T14:25:40.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e580767c-b65f-4126-a98e-f84c273a1fb6' 2024-09-13T14:25:40.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:25:40.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBkS+Rm4dTMJxAAie5XcKKjOttlyYSi1mXQMw== 2024-09-13T14:25:40.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBkS+Rm4dTMJxAAie5XcKKjOttlyYSi1mXQMw=="}' 2024-09-13T14:25:40.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e580767c-b65f-4126-a98e-f84c273a1fb6 -i td/osd-scrub-repair/0/new.json 2024-09-13T14:25:40.902 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:25:40.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T14:25:40.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBkS+Rm4dTMJxAAie5XcKKjOttlyYSi1mXQMw== --osd-uuid e580767c-b65f-4126-a98e-f84c273a1fb6 2024-09-13T14:25:40.958 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:40.957+0000 7f62f81a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:25:40.960 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:40.961+0000 7f62f81a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:25:40.961 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:40.961+0000 7f62f81a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:25:40.962 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:40.961+0000 7f62f81a87c0 -1 bdev(0x561b336cfc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:25:40.962 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:40.961+0000 7f62f81a87c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T14:25:47.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T14:25:47.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:25:47.290 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:25:47.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T14:25:47.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:25:47.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T14:25:47.689 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:25:47.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:25:47.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:25:47.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:25:47.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:25:47.719 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:47.717+0000 7fecf05a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:25:47.721 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:47.721+0000 7fecf05a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:25:47.722 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:47.721+0000 7fecf05a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:25:48.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T14:25:48.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:25:48.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:25:48.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:25:48.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:25:48.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:48.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:25:48.034 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:25:48.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:48.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:25:48.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:49.210 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:49.209+0000 7fecf05a57c0 -1 Falling back to public interface 2024-09-13T14:25:49.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:49.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:49.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:25:49.392 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:25:49.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:49.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:25:49.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:50.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:50.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:50.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:25:50.709 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:25:50.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:50.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:25:51.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:51.944 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:25:51.945+0000 7fecf05a57c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:25:52.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:52.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:52.034 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:25:52.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:25:52.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:52.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:25:52.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:53.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:53.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:53.435 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:25:53.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:25:53.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:53.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:25:53.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:54.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:54.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:54.881 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:25:54.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:25:54.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:54.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:25:55.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:56.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:56.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:56.218 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:25:56.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:25:56.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:56.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:25:56.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:57.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:57.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:57.552 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:25:57.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:25:57.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:57.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:25:57.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:25:58.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:25:58.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:25:58.921 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:25:58.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:25:58.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:25:58.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:25:59.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:00.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:00.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:00.263 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:26:00.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:26:00.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:00.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:26:00.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:01.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:01.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:01.697 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:26:01.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:26:01.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:01.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:26:02.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:03.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:03.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:03.022 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:26:03.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:26:03.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:03.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:26:03.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:04.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:04.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:04.344 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:26:04.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:26:04.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:04.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:26:05.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:06.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:06.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:06.171 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:26:06.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:26:06.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:06.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:26:06.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:07.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:07.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:07.862 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:26:07.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:26:07.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:07.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:26:08.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:09.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:09.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:09.790 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:26:09.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:26:09.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:09.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:26:10.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:11.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:11.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:11.610 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:26:11.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:26:11.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:11.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:26:12.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:13.002 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:26:13.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:26:13.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:13.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:26:13.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:14.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:14.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:14.529 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:26:14.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:26:14.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:14.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:26:14.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:15.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:15.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:15.995 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:26:15.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:26:15.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:15.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:26:16.431 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/782051874,v1:127.0.0.1:6803/782051874] [v2:127.0.0.1:6804/782051874,v1:127.0.0.1:6805/782051874] exists,up e580767c-b65f-4126-a98e-f84c273a1fb6 2024-09-13T14:26:16.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:26:16.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:26:16.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:26:16.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-13T14:26:16.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:26:16.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:26:16.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:26:16.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T14:26:16.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:26:16.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:26:16.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:26:16.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:26:16.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:26:16.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:26:16.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:26:16.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:26:16.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:26:16.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:26:16.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:26:16.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:26:16.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:26:16.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:26:16.436 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:26:16.436 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:26:16.436 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:26:16.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:26:16.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:26:16.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:26:16.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:26:16.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:26:16.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:26:16.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:26:16.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:26:16.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:26:16.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:26:16.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:26:16.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:26:16.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:26:16.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:26:16.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:26:16.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5d0c7c71-f1da-432f-b7ed-46b17df1f3ca 2024-09-13T14:26:16.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 5d0c7c71-f1da-432f-b7ed-46b17df1f3ca' 2024-09-13T14:26:16.438 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 5d0c7c71-f1da-432f-b7ed-46b17df1f3ca 2024-09-13T14:26:16.438 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:26:16.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCIS+RmnSudGxAA3zi135o6RC3BTC76c8LpIQ== 2024-09-13T14:26:16.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCIS+RmnSudGxAA3zi135o6RC3BTC76c8LpIQ=="}' 2024-09-13T14:26:16.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5d0c7c71-f1da-432f-b7ed-46b17df1f3ca -i td/osd-scrub-repair/1/new.json 2024-09-13T14:26:16.825 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:26:16.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T14:26:16.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCIS+RmnSudGxAA3zi135o6RC3BTC76c8LpIQ== --osd-uuid 5d0c7c71-f1da-432f-b7ed-46b17df1f3ca 2024-09-13T14:26:16.875 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:26:16.876+0000 7f2c2bc4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:26:16.910 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:26:16.912+0000 7f2c2bc4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:26:16.911 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:26:16.912+0000 7f2c2bc4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:26:16.912 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:26:16.916+0000 7f2c2bc4c7c0 -1 bdev(0x55c68918a800 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:26:16.912 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:26:16.916+0000 7f2c2bc4c7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T14:26:23.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T14:26:23.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:26:23.547 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:26:23.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T14:26:23.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:26:23.957 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:26:23.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T14:26:23.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:26:23.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:26:23.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:26:23.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:26:23.994 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:26:23.995+0000 7f5ebc4577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:26:24.017 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:26:24.019+0000 7f5ebc4577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:26:24.018 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:26:24.019+0000 7f5ebc4577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:26:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T14:26:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:26:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:26:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:26:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:26:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:26:24.306 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:26:24.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:24.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:24.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:25.099 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:26:25.099+0000 7f5ebc4577c0 -1 Falling back to public interface 2024-09-13T14:26:25.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:25.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:25.646 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:26:25.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:26:25.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:25.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:26.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:27.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:27.037 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:26:27.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:27.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:26:27.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:27.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:27.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:28.168 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:26:28.171+0000 7f5ebc4577c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:26:28.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:28.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:28.460 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:26:28.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:26:28.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:28.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:28.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:29.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:29.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:29.951 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:26:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:26:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:30.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:31.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:31.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:31.352 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:26:31.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:26:31.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:31.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:31.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:32.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:32.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:32.680 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:26:32.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:26:32.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:32.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:33.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:34.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:34.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:34.144 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:26:34.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:26:34.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:34.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:34.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:35.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:35.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:35.509 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:26:35.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:26:35.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:35.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:35.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:36.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:36.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:36.846 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:26:36.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:26:36.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:36.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:37.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:38.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:38.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:38.285 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:26:38.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:26:38.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:38.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:38.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:39.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:39.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:39.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:26:39.634 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:26:39.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:39.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:39.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:40.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:40.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:40.993 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:26:40.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:26:40.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:40.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:41.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:42.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:42.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:42.329 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:26:42.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:26:42.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:42.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:42.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:43.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:43.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:43.715 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:26:43.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:26:43.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:43.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:44.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:45.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:45.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:45.559 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:26:45.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:26:45.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:45.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:46.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:47.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:47.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:47.153 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:26:47.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:26:47.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:47.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:47.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:48.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:48.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:48.923 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:26:48.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:26:48.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:48.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:49.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:50.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:50.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:50.633 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:26:50.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:26:50.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:50.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:51.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:52.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:52.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:52.057 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:26:52.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:26:52.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:52.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:52.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:26:53.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:26:53.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:26:53.563 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T14:26:53.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T14:26:53.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:26:53.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:26:53.891 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2832386020,v1:127.0.0.1:6811/2832386020] [v2:127.0.0.1:6812/2832386020,v1:127.0.0.1:6813/2832386020] exists,up 5d0c7c71-f1da-432f-b7ed-46b17df1f3ca 2024-09-13T14:26:53.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:26:53.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:26:53.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:26:53.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-13T14:26:53.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:26:53.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:26:53.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:26:53.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T14:26:53.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:26:53.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:26:53.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:26:53.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:26:53.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:26:53.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:26:53.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T14:26:53.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T14:26:53.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:26:53.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:26:53.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:26:53.895 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:26:53.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:26:53.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:26:53.896 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:26:53.896 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:26:53.896 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:26:53.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:26:53.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:26:53.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:26:53.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:26:53.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:26:53.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:26:53.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:26:53.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:26:53.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:26:53.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:26:53.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:26:53.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:26:53.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:26:53.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T14:26:53.899 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:26:53.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4360a5e8-7f16-4de7-8f5e-f4976110d97c 2024-09-13T14:26:53.901 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 4360a5e8-7f16-4de7-8f5e-f4976110d97c 2024-09-13T14:26:53.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 4360a5e8-7f16-4de7-8f5e-f4976110d97c' 2024-09-13T14:26:53.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:26:53.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCtS+RmCPT2NhAAoA6YPfLSrEAdozYHiv6oSA== 2024-09-13T14:26:53.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCtS+RmCPT2NhAAoA6YPfLSrEAdozYHiv6oSA=="}' 2024-09-13T14:26:53.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4360a5e8-7f16-4de7-8f5e-f4976110d97c -i td/osd-scrub-repair/2/new.json 2024-09-13T14:26:54.479 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:26:54.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T14:26:54.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCtS+RmCPT2NhAAoA6YPfLSrEAdozYHiv6oSA== --osd-uuid 4360a5e8-7f16-4de7-8f5e-f4976110d97c 2024-09-13T14:26:54.538 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:26:54.538+0000 7f9478f707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:26:54.581 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:26:54.582+0000 7f9478f707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:26:54.583 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:26:54.582+0000 7f9478f707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:26:54.583 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:26:54.582+0000 7f9478f707c0 -1 bdev(0x56089a8c3c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T14:26:54.583 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:26:54.582+0000 7f9478f707c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T14:27:02.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T14:27:02.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:27:02.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T14:27:02.057 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T14:27:02.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:27:02.455 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T14:27:02.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T14:27:02.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:27:02.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:27:02.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:27:02.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:27:02.492 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:27:02.490+0000 7f8ea03cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:27:02.517 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:27:02.518+0000 7f8ea03cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:27:02.518 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:27:02.518+0000 7f8ea03cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:27:02.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T14:27:02.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:27:02.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:27:02.794 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:27:02.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:27:02.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:27:02.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:02.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:27:02.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:02.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:03.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:04.051 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:27:04.050+0000 7f8ea03cc7c0 -1 Falling back to public interface 2024-09-13T14:27:04.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:04.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:04.161 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:27:04.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:27:04.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:04.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:04.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:05.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:05.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:05.500 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:27:05.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:27:05.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:05.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:05.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:06.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:06.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:06.841 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:27:06.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:27:06.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:06.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:06.953 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:27:06.954+0000 7f8ea03cc7c0 -1 osd.2 0 log_to_monitors true 2024-09-13T14:27:07.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:08.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:08.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:08.214 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:27:08.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:27:08.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:08.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:08.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:09.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:09.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:09.538 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:27:09.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:27:09.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:09.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:10.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:11.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:11.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:11.037 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:27:11.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:27:11.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:11.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:11.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:12.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:12.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:12.526 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:27:12.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:27:12.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:12.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:12.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:13.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:13.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:13.855 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:27:13.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:27:13.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:13.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:14.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:15.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:15.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:15.202 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:27:15.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:27:15.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:15.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:15.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:16.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:16.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:16.632 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:27:16.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:27:16.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:16.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:17.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:18.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:18.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:18.036 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:27:18.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:27:18.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:18.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:18.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:19.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:19.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:19.356 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:27:19.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:27:19.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:19.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:19.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:20.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:20.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:20.676 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:27:20.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:27:20.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:20.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:21.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:22.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:22.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:22.106 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:27:22.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:27:22.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:22.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:22.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:23.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:23.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:23.885 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:27:23.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:27:23.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:23.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:24.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:25.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:25.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:25.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:27:25.294 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:27:25.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:25.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:25.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:26.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:26.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:26.703 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:27:26.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:27:26.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:26.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:27.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:28.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:28.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:28.133 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:27:28.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:27:28.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:28.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:28.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:29.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:29.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:29.816 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:27:29.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:27:29.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:29.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:30.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:31.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:31.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:31.262 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T14:27:31.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T14:27:31.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:31.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:32.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:33.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:33.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:33.157 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T14:27:33.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T14:27:33.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:33.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:27:33.492 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1843837483,v1:127.0.0.1:6819/1843837483] [v2:127.0.0.1:6820/1843837483,v1:127.0.0.1:6821/1843837483] exists,up 4360a5e8-7f16-4de7-8f5e-f4976110d97c 2024-09-13T14:27:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:27:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:27:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:27:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-09-13T14:27:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T14:27:33.812 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' does not exist 2024-09-13T14:27:33.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T14:27:33.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T14:27:34.245 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T14:27:34.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:27:35.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T14:27:37.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-09-13T14:27:37.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:27:37.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:27:37.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:27:37.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:27:37.607 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:27:37.607 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:27:37.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:27:37.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:27:37.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:27:37.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:27:37.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:27:37.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:27:37.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:27:37.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:27:37.737 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:27:38.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:27:38.118 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:27:38.118 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:27:38.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:27:38.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:27:38.119 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:27:38.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-13T14:27:38.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-13T14:27:38.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-13T14:27:38.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:27:38.278 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:27:38.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=68719476750 2024-09-13T14:27:38.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 68719476750 2024-09-13T14:27:38.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-68719476750' 2024-09-13T14:27:38.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:27:38.422 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:27:38.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822663 2024-09-13T14:27:38.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822663 2024-09-13T14:27:38.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-68719476750 2-154618822663' 2024-09-13T14:27:38.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:27:38.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-13T14:27:38.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:27:38.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:27:38.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-13T14:27:38.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:27:38.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-13T14:27:38.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-13T14:27:38.570 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836507 2024-09-13T14:27:38.570 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:27:38.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836507 2024-09-13T14:27:38.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:27:39.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:27:39.892 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:27:40.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-13T14:27:40.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:27:40.258 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-68719476750 2024-09-13T14:27:40.258 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:27:40.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:27:40.260 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-68719476750 2024-09-13T14:27:40.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:27:40.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=68719476750 2024-09-13T14:27:40.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 68719476750' 2024-09-13T14:27:40.262 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 68719476750 2024-09-13T14:27:40.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:27:40.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 68719476751 -lt 68719476750 2024-09-13T14:27:40.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:27:40.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-154618822663 2024-09-13T14:27:40.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:27:40.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:27:40.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-154618822663 2024-09-13T14:27:40.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:27:40.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822663 2024-09-13T14:27:40.589 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 154618822663 2024-09-13T14:27:40.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 154618822663' 2024-09-13T14:27:40.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:27:40.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822663 -lt 154618822663 2024-09-13T14:27:40.921 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:27:40.922 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:27:40.922 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:27:41.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T14:27:41.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:27:41.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:27:41.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:27:41.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:27:41.521 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:27:41.521 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:27:41.521 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:27:41.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:27:41.845 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:27:41.845 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:27:41.845 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:27:42.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T14:27:42.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:27:42.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:27:42.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-09-13T14:27:42.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-13T14:27:42.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-13T14:27:42.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-13T14:27:42.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-13T14:27:42.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-13T14:27:42.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-13T14:27:42.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-13T14:27:43.632 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'ecpool' created 2024-09-13T14:27:43.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:27:44.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-13T14:27:44.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-13T14:27:45.257 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 2 allow_ec_overwrites to true 2024-09-13T14:27:45.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-13T14:27:45.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:27:45.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:27:45.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:27:45.283 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:27:45.284 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:27:45.285 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:27:45.285 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:27:45.285 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:27:45.285 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:27:45.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:27:45.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:27:45.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:27:45.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:27:45.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:27:45.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:27:45.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:27:45.823 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:27:45.823 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:27:45.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:27:45.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:27:45.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:27:45.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836509 2024-09-13T14:27:45.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836509 2024-09-13T14:27:45.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509' 2024-09-13T14:27:45.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:27:45.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:27:46.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=68719476753 2024-09-13T14:27:46.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 68719476753 2024-09-13T14:27:46.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-68719476753' 2024-09-13T14:27:46.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:27:46.151 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:27:46.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822665 2024-09-13T14:27:46.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822665 2024-09-13T14:27:46.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-68719476753 2-154618822665' 2024-09-13T14:27:46.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:27:46.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836509 2024-09-13T14:27:46.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:27:46.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:27:46.269 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836509 2024-09-13T14:27:46.269 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:27:46.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836509 2024-09-13T14:27:46.270 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836509 2024-09-13T14:27:46.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-13T14:27:46.270 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:27:46.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836509 2024-09-13T14:27:46.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:27:47.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:27:47.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:27:47.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836509 2024-09-13T14:27:47.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:27:47.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-68719476753 2024-09-13T14:27:47.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:27:47.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:27:47.947 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:27:47.947 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-68719476753 2024-09-13T14:27:47.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=68719476753 2024-09-13T14:27:47.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 68719476753' 2024-09-13T14:27:47.949 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 68719476753 2024-09-13T14:27:47.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:27:48.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 68719476753 -lt 68719476753 2024-09-13T14:27:48.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:27:48.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-154618822665 2024-09-13T14:27:48.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:27:48.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:27:48.339 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:27:48.339 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-154618822665 2024-09-13T14:27:48.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822665 2024-09-13T14:27:48.340 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 154618822665 2024-09-13T14:27:48.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 154618822665' 2024-09-13T14:27:48.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:27:48.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822665 -lt 154618822665 2024-09-13T14:27:48.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:27:48.665 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:27:48.665 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:27:49.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:27:49.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:27:49.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:27:49.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:27:49.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:27:49.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:27:49.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:27:49.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:27:49.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:27:49.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:27:49.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:27:49.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:27:49.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:27:49.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:27:49.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:27:49.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-13T14:27:49.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-09-13T14:27:49.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-09-13T14:27:49.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-13T14:27:49.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-13T14:27:49.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-13T14:27:49.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:27:49.978 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-13T14:27:49.978 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-13T14:27:49.978 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:27:49.978 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:27:49.978 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:27:50.334 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-13T14:27:50.334 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:27:50.334 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-13T14:27:50.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:27:50.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:27:50.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:27:50.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:27:50.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:27:50.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:27:50.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:27:50.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:27:50.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:27:50.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:27:50.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:27:50.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:27:50.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:27:50.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:27:50.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:27:50.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:27:51.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:27:51.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:27:51.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:27:51.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:27:51.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:27:51.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:27:51.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:27:51.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-13T14:27:52.994 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-13T14:27:52.994 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-13T14:27:52.994 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-13T14:27:53.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:27:53.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:27:53.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:27:53.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:27:53.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:27:53.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:27:53.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:27:53.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:27:53.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:27:53.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:27:53.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:27:53.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:27:53.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:27:53.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:27:53.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:27:53.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:27:53.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:27:53.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:27:53.534 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:27:53.535 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:27:53.535 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:27:53.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:27:53.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:27:53.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:27:53.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:27:53.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:27:53.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:27:53.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:27:53.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:27:53.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:27:53.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:27:53.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:27:53.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:27:53.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:27:53.536 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T14:27:53.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:27:53.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:27:53.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:27:53.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:27:53.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:27:53.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:27:53.575 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:27:53.573+0000 7f156c2107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:27:53.619 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:27:53.617+0000 7f156c2107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:27:53.620 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:27:53.617+0000 7f156c2107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:27:53.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:27:53.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:27:53.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:27:53.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:27:53.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:27:53.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:53.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:27:53.861 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:27:53.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:53.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:27:54.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:54.879 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:27:54.877+0000 7f156c2107c0 -1 Falling back to public interface 2024-09-13T14:27:55.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:55.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:55.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:27:55.210 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:27:55.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:55.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:27:55.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:56.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:56.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:56.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:27:56.565 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:27:56.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:56.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:27:56.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:57.828 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:27:57.824+0000 7f156c2107c0 -1 osd.1 58 log_to_monitors true 2024-09-13T14:27:57.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:57.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:57.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:27:57.901 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:27:57.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:57.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:27:58.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:27:59.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:27:59.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:27:59.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:27:59.306 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:27:59.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:27:59.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:27:59.786 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 66 up_thru 66 down_at 60 last_clean_interval [16,58) [v2:127.0.0.1:6810/3608299620,v1:127.0.0.1:6811/3608299620] [v2:127.0.0.1:6812/3608299620,v1:127.0.0.1:6813/3608299620] exists,up 5d0c7c71-f1da-432f-b7ed-46b17df1f3ca 2024-09-13T14:27:59.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:27:59.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:27:59.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:27:59.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:27:59.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:27:59.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:27:59.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:27:59.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:27:59.787 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:27:59.787 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:27:59.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:27:59.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:27:59.788 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:27:59.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:27:59.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:27:59.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:27:59.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:27:59.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:27:59.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:28:00.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:28:00.368 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:28:00.368 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:28:00.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:28:00.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:28:00.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:28:00.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836513 2024-09-13T14:28:00.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836513 2024-09-13T14:28:00.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513' 2024-09-13T14:28:00.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:28:00.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:28:00.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841538 2024-09-13T14:28:00.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841538 2024-09-13T14:28:00.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-283467841538' 2024-09-13T14:28:00.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:28:00.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:28:00.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822669 2024-09-13T14:28:00.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822669 2024-09-13T14:28:00.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-283467841538 2-154618822669' 2024-09-13T14:28:00.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:28:00.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836513 2024-09-13T14:28:00.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:28:00.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:28:00.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836513 2024-09-13T14:28:00.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:28:00.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836513 2024-09-13T14:28:00.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-13T14:28:00.803 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836513 2024-09-13T14:28:00.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:28:01.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836513 2024-09-13T14:28:01.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:28:02.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:28:02.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:28:02.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836513 -lt 21474836513 2024-09-13T14:28:02.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:28:02.460 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:28:02.460 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-283467841538 2024-09-13T14:28:02.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:28:02.462 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-283467841538 2024-09-13T14:28:02.462 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:28:02.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841538 2024-09-13T14:28:02.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 283467841538' 2024-09-13T14:28:02.464 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 283467841538 2024-09-13T14:28:02.464 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:28:02.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841538 -lt 283467841538 2024-09-13T14:28:02.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:28:02.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-154618822669 2024-09-13T14:28:02.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:28:02.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:28:02.786 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-154618822669 2024-09-13T14:28:02.786 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:28:02.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822669 2024-09-13T14:28:02.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 154618822669' 2024-09-13T14:28:02.788 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 154618822669 2024-09-13T14:28:02.788 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:28:03.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822669 -lt 154618822669 2024-09-13T14:28:03.122 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:28:03.122 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:28:03.122 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:28:03.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:28:03.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:28:03.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:28:03.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:28:03.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:28:03.540 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:28:03.540 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:28:03.540 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:28:03.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:28:03.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:28:03.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:28:03.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:28:04.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:28:04.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:28:04.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:28:04.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-09-13T14:28:04.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:28:04.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:28:04.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:28:04.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:28:04.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-09-13T14:28:04.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-09-13T14:28:04.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:28:04.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:28:04.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:28:04.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:28:04.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-09-13T14:27:43.476407+0000 2024-09-13T14:28:04.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:28:04.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:27:43.476407+0000 2024-09-13T14:28:04.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:28:04.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:28:04.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:28:04.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:28:04.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:28:04.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:28:04.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:28:04.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:28:05.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:27:43.476407+0000 '>' 2024-09-13T14:27:43.476407+0000 2024-09-13T14:28:05.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:28:06.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:28:06.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:28:06.282 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:28:06.282 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:28:06.282 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:28:06.282 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:28:06.283 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:28:06.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:27:43.476407+0000 '>' 2024-09-13T14:27:43.476407+0000 2024-09-13T14:28:06.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:28:07.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:28:07.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:28:07.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:28:07.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:28:07.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:28:07.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:28:07.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:28:07.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:27:43.476407+0000 '>' 2024-09-13T14:27:43.476407+0000 2024-09-13T14:28:07.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:28:08.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:28:08.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:28:08.998 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:28:08.998 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:28:08.998 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:28:08.999 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:28:08.999 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:28:09.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:28:03.837083+0000 '>' 2024-09-13T14:27:43.476407+0000 2024-09-13T14:28:09.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:28:09.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-09-13T14:28:09.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:28:09.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:28:09.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:28:09.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:28:09.330 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:28:09.330 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:28:09.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:28:09.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:28:09.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:28:09.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:28:09.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:28:09.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:28:09.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:28:09.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:28:09.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:28:09.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:28:09.763 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:28:09.763 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:28:09.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:28:09.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:28:09.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:28:09.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836516 2024-09-13T14:28:09.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836516 2024-09-13T14:28:09.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516' 2024-09-13T14:28:09.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:28:09.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:28:10.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841541 2024-09-13T14:28:10.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841541 2024-09-13T14:28:10.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516 1-283467841541' 2024-09-13T14:28:10.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:28:10.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:28:10.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822672 2024-09-13T14:28:10.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822672 2024-09-13T14:28:10.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516 1-283467841541 2-154618822672' 2024-09-13T14:28:10.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:28:10.205 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836516 2024-09-13T14:28:10.205 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:28:10.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:28:10.207 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836516 2024-09-13T14:28:10.207 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:28:10.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836516 2024-09-13T14:28:10.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-09-13T14:28:10.208 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836516 2024-09-13T14:28:10.208 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:28:10.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836515 -lt 21474836516 2024-09-13T14:28:10.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:28:11.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:28:11.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:28:11.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836516 -lt 21474836516 2024-09-13T14:28:11.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:28:11.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-283467841541 2024-09-13T14:28:11.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:28:11.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:28:11.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:28:11.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-283467841541 2024-09-13T14:28:11.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841541 2024-09-13T14:28:11.854 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 283467841541 2024-09-13T14:28:11.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 283467841541' 2024-09-13T14:28:11.855 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:28:12.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841541 -lt 283467841541 2024-09-13T14:28:12.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:28:12.190 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-154618822672 2024-09-13T14:28:12.190 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:28:12.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:28:12.193 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-154618822672 2024-09-13T14:28:12.193 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:28:12.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822672 2024-09-13T14:28:12.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 154618822672' 2024-09-13T14:28:12.195 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 154618822672 2024-09-13T14:28:12.195 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:28:12.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822672 -lt 154618822672 2024-09-13T14:28:12.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:28:12.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:28:12.603 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:28:12.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:28:12.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:28:12.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:28:12.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:28:12.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:28:12.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:28:12.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:28:12.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:28:13.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:28:13.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:28:13.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:28:13.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:28:13.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:28:13.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:28:13.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:28:13.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-13T14:28:13.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-13T14:28:13.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:28:13.710 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-13T14:28:13.710 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-13T14:28:13.710 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:28:13.710 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:28:13.710 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:28:14.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-13T14:28:14.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:28:14.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-13T14:28:14.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:28:14.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:28:14.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:28:14.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:28:14.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:28:14.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:28:14.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:28:14.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:28:14.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:28:14.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:28:14.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:28:14.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:28:14.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:28:14.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:28:14.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:28:14.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:28:14.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:28:14.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:28:14.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:28:14.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:28:14.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:28:14.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:28:14.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:28:14.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-13T14:28:15.313 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-13T14:28:15.313 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-13T14:28:15.313 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:28:15.314 INFO:tasks.workunit.client.0.smithi112.stdout:hinfo_key 2024-09-13T14:28:15.314 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:28:15.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:28:15.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:28:15.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:28:15.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:28:15.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:28:15.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:28:15.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:28:15.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:28:15.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:28:15.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:28:15.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:28:15.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:28:15.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:28:15.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:28:15.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:28:15.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:28:15.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:28:15.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:28:15.602 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:28:15.602 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:28:15.602 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:28:15.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:28:15.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:28:15.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:28:15.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:28:15.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:28:15.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:28:15.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:28:15.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:28:15.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:28:15.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:28:15.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T14:28:15.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:28:15.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:28:15.604 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T14:28:15.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-13T14:28:15.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:28:15.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:28:15.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:28:15.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:28:15.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:28:15.643 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:28:15.644+0000 7fde6b5527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:28:15.716 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:28:15.720+0000 7fde6b5527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:28:15.718 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:28:15.720+0000 7fde6b5527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:28:15.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:28:15.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:28:15.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:28:15.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:28:15.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:28:15.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:15.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:28:15.998 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:28:15.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:15.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:28:16.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:17.272 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:28:17.272+0000 7fde6b5527c0 -1 Falling back to public interface 2024-09-13T14:28:17.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:17.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:17.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:28:17.323 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:28:17.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:17.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:28:17.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:18.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:18.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:18.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:28:18.655 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:28:18.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:18.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:28:18.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:19.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:19.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:19.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:28:19.985 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:28:19.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:19.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:28:20.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:20.702 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:28:20.704+0000 7fde6b5527c0 -1 osd.1 76 log_to_monitors true 2024-09-13T14:28:21.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:21.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:21.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:28:21.323 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:28:21.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:21.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:28:21.792 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 83 up_thru 83 down_at 77 last_clean_interval [66,76) [v2:127.0.0.1:6810/386228265,v1:127.0.0.1:6811/386228265] [v2:127.0.0.1:6812/386228265,v1:127.0.0.1:6813/386228265] exists,up 5d0c7c71-f1da-432f-b7ed-46b17df1f3ca 2024-09-13T14:28:21.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:28:21.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:28:21.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:28:21.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:28:21.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:28:21.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:28:21.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:28:21.793 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:28:21.794 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:28:21.794 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:28:21.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:28:21.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:28:21.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:28:21.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:28:21.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:28:21.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:28:21.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:28:21.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:28:21.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:28:22.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:28:22.301 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:28:22.302 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T14:28:22.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:28:22.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:28:22.302 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:28:22.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836520 2024-09-13T14:28:22.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836520 2024-09-13T14:28:22.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520' 2024-09-13T14:28:22.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:28:22.636 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:28:22.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=356482285570 2024-09-13T14:28:22.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 356482285570 2024-09-13T14:28:22.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520 1-356482285570' 2024-09-13T14:28:22.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:28:22.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:28:23.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822675 2024-09-13T14:28:23.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822675 2024-09-13T14:28:23.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520 1-356482285570 2-154618822675' 2024-09-13T14:28:23.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:28:23.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836520 2024-09-13T14:28:23.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:28:23.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:28:23.068 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836520 2024-09-13T14:28:23.068 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:28:23.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836520 2024-09-13T14:28:23.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-09-13T14:28:23.069 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836520 2024-09-13T14:28:23.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:28:23.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836520 -lt 21474836520 2024-09-13T14:28:23.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:28:23.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-356482285570 2024-09-13T14:28:23.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:28:23.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:28:23.458 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-356482285570 2024-09-13T14:28:23.459 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:28:23.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=356482285570 2024-09-13T14:28:23.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 356482285570' 2024-09-13T14:28:23.460 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 356482285570 2024-09-13T14:28:23.460 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:28:23.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 356482285570 -lt 356482285570 2024-09-13T14:28:23.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:28:23.821 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-154618822675 2024-09-13T14:28:23.821 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:28:23.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:28:23.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-154618822675 2024-09-13T14:28:23.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:28:23.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822675 2024-09-13T14:28:23.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 154618822675' 2024-09-13T14:28:23.826 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 154618822675 2024-09-13T14:28:23.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:28:24.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822675 -lt 154618822675 2024-09-13T14:28:24.142 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:28:24.143 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:28:24.143 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:28:24.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:28:24.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:28:24.550 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:28:24.550 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:28:24.550 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:28:24.550 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:28:24.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:28:24.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:28:24.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:28:24.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:28:24.872 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:28:24.872 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:28:25.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:28:25.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:28:25.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:28:25.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:28:25.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:28:25.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T14:28:25.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:28:25.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:28:25.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:28:25.302 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:28:25.302 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:28:25.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:28:25.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:28:25.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:28:25.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:28:25.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:28:25.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:28:25.613 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:28:25.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:28:25.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:28:25.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:28:25.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:28:25.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:28:25.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:28:25.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:28:25.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:28:25.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:28:25.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:28:25.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:28:25.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:28:25.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:28:25.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:28:25.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:28:25.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:28:25.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T14:28:25.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T14:28:25.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T14:28:25.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T14:28:25.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:28:25.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:28:25.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:28:25.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:28:25.647 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:28:25.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:28:25.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:28:25.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:28:25.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:28:25.649 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:28:25.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:28:25.650 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:28:25.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:28:25.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:28:25.652 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:28:25.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:28:25.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:28:25.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:28:25.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:28:25.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:28:25.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:28:25.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:28:25.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:28:25.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:28:25.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:28:25.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:28:25.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:28:25.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:28:25.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T14:28:25.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:28:25.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:28:25.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:28:25.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T14:28:25.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:28:25.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T14:28:25.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:28:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T14:28:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-09-13T14:28:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-09-13T14:28:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-09-13T14:28:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-09-13T14:28:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-09-13T14:28:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-09-13T14:28:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T14:28:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T14:28:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T14:28:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T14:28:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T14:28:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T14:28:26.162 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T14:28:26.162 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:28:26.162 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:28:26.163 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:28:26.163 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:28:26.163 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:28:26.163 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:28:26.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T14:28:26.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T14:28:26.366 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T14:28:26.366 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:28:26.366 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:28:26.366 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T14:28:26.366 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T14:28:26.367 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:28:26.367 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:28:26.367 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:28:26.368 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:28:26.368 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:28:26.368 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:28:26.368 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:28:26.368 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:28:26.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T14:28:26.485 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T14:28:26.485 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:28:26.485 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:28:26.485 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T14:28:26.485 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T14:28:26.486 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:28:26.487 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:28:26.487 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:28:26.487 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:28:26.487 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:28:26.487 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:28:26.487 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:28:26.487 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:28:26.487 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T14:28:26.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-09-13T14:28:26.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T14:28:26.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T14:28:26.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T14:28:26.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T14:28:26.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T14:28:26.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T14:28:27.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T14:28:27.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:28:27.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:28:27.484 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:28:27.484 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:28:27.484 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:28:27.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:28:27.485 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:28:27.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:28:27.521 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-09-13T14:28:27.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-13T14:28:27.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-09-13T14:28:27.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:28:27.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:28:27.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T14:28:27.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:28:27.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:28:27.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:28:27.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:28:27.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:28:27.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:28:27.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:28:27.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:28:27.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:28:27.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:28:27.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:28:27.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:28:27.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:28:27.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:28:27.526 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:28:27.526 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:28:27.526 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:28:27.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:28:27.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:28:27.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:28:27.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:28:27.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:28:27.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:28:27.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:28:27.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:28:27.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:28:27.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:28:27.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:28:27.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:28:27.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:28:27.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:28:27.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:28:27.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=04fc5724-a982-4b7e-ad2e-9a46b56a2480 2024-09-13T14:28:27.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 04fc5724-a982-4b7e-ad2e-9a46b56a2480' 2024-09-13T14:28:27.529 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 04fc5724-a982-4b7e-ad2e-9a46b56a2480 2024-09-13T14:28:27.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:28:27.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQALTORm3xirIBAAorgJCb+99KRd5QfKrtMFbQ== 2024-09-13T14:28:27.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQALTORm3xirIBAAorgJCb+99KRd5QfKrtMFbQ=="}' 2024-09-13T14:28:27.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 04fc5724-a982-4b7e-ad2e-9a46b56a2480 -i td/osd-scrub-repair/0/new.json 2024-09-13T14:28:27.818 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:28:27.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T14:28:27.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALTORm3xirIBAAorgJCb+99KRd5QfKrtMFbQ== --osd-uuid 04fc5724-a982-4b7e-ad2e-9a46b56a2480 2024-09-13T14:28:27.873 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:28:27.876+0000 7f286fdd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:28:27.885 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:28:27.888+0000 7f286fdd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:28:27.887 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:28:27.888+0000 7f286fdd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:28:27.887 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:28:27.888+0000 7f286fdd77c0 -1 bdev(0x559a74445c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:28:27.887 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:28:27.888+0000 7f286fdd77c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T14:28:34.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T14:28:34.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:28:34.325 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:28:34.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T14:28:34.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:28:34.733 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:28:34.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T14:28:34.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:28:34.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:28:34.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:28:34.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:28:34.766 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:28:34.767+0000 7f209155c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:28:34.792 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:28:34.791+0000 7f209155c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:28:34.793 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:28:34.795+0000 7f209155c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:28:35.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T14:28:35.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:28:35.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:28:35.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:28:35.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:28:35.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:35.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:28:35.074 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:28:35.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:35.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:35.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:36.091 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:28:36.091+0000 7f209155c7c0 -1 Falling back to public interface 2024-09-13T14:28:36.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:36.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:36.411 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:28:36.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:28:36.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:36.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:36.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:37.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:37.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:37.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:28:37.731 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:28:37.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:37.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:38.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:38.886 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:28:38.887+0000 7f209155c7c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:28:39.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:39.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:39.065 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:28:39.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:28:39.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:39.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:39.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:40.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:40.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:40.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:28:40.567 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:28:40.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:40.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:40.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:41.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:41.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:41.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:28:41.905 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:28:41.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:41.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:42.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:43.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:43.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:43.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:28:43.231 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:28:43.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:43.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:43.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:44.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:44.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:44.689 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:28:44.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:28:44.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:44.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:45.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:46.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:46.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:46.026 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:28:46.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:28:46.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:46.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:46.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:47.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:47.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:47.344 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:28:47.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:28:47.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:47.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:47.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:48.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:48.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:48.666 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:28:48.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:28:48.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:48.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:48.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:49.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:49.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:49.983 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:28:49.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:28:49.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:49.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:50.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:51.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:51.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:51.323 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:28:51.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:28:51.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:51.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:52.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:53.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:53.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:53.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:28:53.053 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:28:53.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:53.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:53.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:54.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:54.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:54.679 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:28:54.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:28:54.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:54.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:55.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:56.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:56.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:56.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:28:56.070 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:28:56.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:56.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:56.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:57.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:57.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:57.536 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:28:57.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:28:57.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:57.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:58.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:28:59.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:28:59.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:28:59.195 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:28:59.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:28:59.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:28:59.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:28:59.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:00.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:00.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:00.975 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:29:00.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:29:00.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:00.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:29:01.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:02.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:02.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:02.607 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:29:02.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:29:02.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:02.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:29:02.988 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4107233570,v1:127.0.0.1:6803/4107233570] [v2:127.0.0.1:6804/4107233570,v1:127.0.0.1:6805/4107233570] exists,up 04fc5724-a982-4b7e-ad2e-9a46b56a2480 2024-09-13T14:29:02.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:29:02.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:29:02.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:29:02.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-13T14:29:02.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-09-13T14:29:02.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:29:02.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:29:02.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T14:29:02.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:29:02.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:29:02.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:29:02.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:29:02.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:29:02.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:29:02.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:29:02.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:29:02.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:29:02.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:29:02.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:29:02.993 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:29:02.993 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:29:02.993 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:29:02.993 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:29:02.993 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:29:02.993 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:29:02.993 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:29:02.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:29:02.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:29:02.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:29:02.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:29:02.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:29:02.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:29:02.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:29:02.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:29:02.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:29:02.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:29:02.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:29:02.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:29:02.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:29:02.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:29:02.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4b91e96b-6165-48ee-b271-78e6091f8314 2024-09-13T14:29:02.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 4b91e96b-6165-48ee-b271-78e6091f8314' 2024-09-13T14:29:02.995 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 4b91e96b-6165-48ee-b271-78e6091f8314 2024-09-13T14:29:02.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:29:03.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAvTORmUQ3oABAAbnbVR4yr48bq1eoaKxERaA== 2024-09-13T14:29:03.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAvTORmUQ3oABAAbnbVR4yr48bq1eoaKxERaA=="}' 2024-09-13T14:29:03.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4b91e96b-6165-48ee-b271-78e6091f8314 -i td/osd-scrub-repair/1/new.json 2024-09-13T14:29:03.456 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:29:03.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T14:29:03.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvTORmUQ3oABAAbnbVR4yr48bq1eoaKxERaA== --osd-uuid 4b91e96b-6165-48ee-b271-78e6091f8314 2024-09-13T14:29:03.501 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:03.503+0000 7f2d59b9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:29:03.540 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:03.539+0000 7f2d59b9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:29:03.542 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:03.543+0000 7f2d59b9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:29:03.542 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:03.543+0000 7f2d59b9d7c0 -1 bdev(0x562442321c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:29:03.542 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:03.543+0000 7f2d59b9d7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T14:29:10.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T14:29:10.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:29:10.539 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:29:10.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T14:29:10.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:29:10.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T14:29:10.938 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:29:10.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:29:10.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:29:10.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:29:10.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:29:10.965 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:10.966+0000 7fa14d5d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:29:10.967 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:10.966+0000 7fa14d5d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:29:10.968 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:10.970+0000 7fa14d5d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:29:11.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T14:29:11.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:29:11.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:29:11.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:29:11.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:29:11.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:11.259 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:29:11.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:29:11.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:11.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:11.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:12.443 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:12.442+0000 7fa14d5d37c0 -1 Falling back to public interface 2024-09-13T14:29:12.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:12.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:12.607 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:29:12.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:29:12.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:12.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:12.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:13.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:13.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:13.944 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:29:13.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:29:13.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:13.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:14.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:15.244 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:15.242+0000 7fa14d5d37c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:29:15.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:15.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:15.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:29:15.284 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:29:15.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:15.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:15.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:16.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:16.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:16.776 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:29:16.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:29:16.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:16.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:17.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:18.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:18.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:18.163 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:29:18.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:29:18.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:18.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:18.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:19.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:19.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:19.513 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:29:19.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:29:19.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:19.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:19.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:20.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:20.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:20.905 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:29:20.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:29:20.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:20.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:21.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:22.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:22.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:22.287 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:29:22.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:29:22.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:22.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:22.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:23.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:23.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:23.683 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:29:23.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:29:23.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:23.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:24.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:25.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:25.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:25.104 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:29:25.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:29:25.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:25.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:25.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:26.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:26.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:26.436 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:29:26.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:29:26.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:26.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:26.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:27.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:27.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:27.908 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:29:27.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:29:27.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:27.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:28.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:29.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:29.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:29.298 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:29:29.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:29:29.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:29.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:29.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:30.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:30.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:30.926 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:29:30.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:29:30.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:30.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:31.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:32.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:32.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:32.802 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:29:32.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:29:32.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:32.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:33.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:34.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:34.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:34.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:29:34.453 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:29:34.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:34.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:35.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:36.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:36.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:36.298 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:29:36.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:29:36.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:36.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:36.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:37.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:37.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:37.671 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:29:37.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:29:37.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:37.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:38.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:39.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:39.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:39.227 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:29:39.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:29:39.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:39.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:39.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:40.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:40.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:40.661 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T14:29:40.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T14:29:40.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:40.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:29:40.983 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4034768813,v1:127.0.0.1:6811/4034768813] [v2:127.0.0.1:6812/4034768813,v1:127.0.0.1:6813/4034768813] exists,up 4b91e96b-6165-48ee-b271-78e6091f8314 2024-09-13T14:29:40.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:29:40.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:29:40.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:29:40.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-13T14:29:40.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-09-13T14:29:40.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:29:40.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:29:40.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T14:29:40.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:29:40.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:29:40.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:29:40.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:29:40.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:29:40.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:29:40.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T14:29:40.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T14:29:40.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:29:40.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:29:40.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:29:40.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:29:40.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:29:40.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:29:40.988 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:29:40.988 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:29:40.988 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:29:40.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:29:40.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:29:40.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:29:40.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:29:40.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:29:40.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:29:40.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:29:40.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:29:40.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:29:40.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:29:40.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:29:40.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:29:40.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:29:40.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T14:29:40.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:29:40.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=aec348e5-2271-4288-b27e-ceb84f9d1513 2024-09-13T14:29:40.990 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 aec348e5-2271-4288-b27e-ceb84f9d1513 2024-09-13T14:29:40.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 aec348e5-2271-4288-b27e-ceb84f9d1513' 2024-09-13T14:29:40.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:29:41.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBVTORmO+/nABAAq/Nix1AeGfS662tURQ4+nA== 2024-09-13T14:29:41.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBVTORmO+/nABAAq/Nix1AeGfS662tURQ4+nA=="}' 2024-09-13T14:29:41.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new aec348e5-2271-4288-b27e-ceb84f9d1513 -i td/osd-scrub-repair/2/new.json 2024-09-13T14:29:41.402 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:29:41.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T14:29:41.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBVTORmO+/nABAAq/Nix1AeGfS662tURQ4+nA== --osd-uuid aec348e5-2271-4288-b27e-ceb84f9d1513 2024-09-13T14:29:41.454 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:41.453+0000 7fe2bb4587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:29:41.496 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:41.493+0000 7fe2bb4587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:29:41.498 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:41.497+0000 7fe2bb4587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:29:41.498 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:41.497+0000 7fe2bb4587c0 -1 bdev(0x559a640c7c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T14:29:41.498 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:41.497+0000 7fe2bb4587c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T14:29:47.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T14:29:47.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:29:47.992 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T14:29:47.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T14:29:47.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:29:48.399 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T14:29:48.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T14:29:48.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:29:48.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:29:48.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:29:48.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:29:48.428 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:48.425+0000 7f67c319d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:29:48.430 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:48.429+0000 7f67c319d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:29:48.431 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:48.429+0000 7f67c319d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:29:48.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T14:29:48.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:29:48.725 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:29:48.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:29:48.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:29:48.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:29:48.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:48.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:29:48.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:48.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:29:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:49.727 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:49.725+0000 7f67c319d7c0 -1 Falling back to public interface 2024-09-13T14:29:50.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:50.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:50.082 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:29:50.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:29:50.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:50.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:29:50.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:51.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:51.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:51.414 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:29:51.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:29:51.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:51.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:29:51.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:52.516 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:29:52.513+0000 7f67c319d7c0 -1 osd.2 0 log_to_monitors true 2024-09-13T14:29:52.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:52.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:52.737 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:29:52.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:29:52.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:52.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:29:53.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:54.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:54.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:54.215 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:29:54.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:29:54.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:54.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:29:54.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:55.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:55.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:55.657 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:29:55.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:29:55.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:55.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:29:55.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:56.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:56.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:56.980 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:29:56.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:29:56.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:56.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:29:57.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:58.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:58.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:58.395 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:29:58.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:29:58.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:58.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:29:58.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:29:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:29:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:29:59.720 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:29:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:29:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:29:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:30:00.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:01.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:01.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:01.059 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:30:01.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:30:01.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:01.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:30:01.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:02.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:02.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:02.378 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:30:02.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:30:02.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:02.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:30:02.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:03.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:03.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:03.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:30:03.849 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:30:03.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:03.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:30:04.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:05.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:05.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:05.185 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:30:05.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:30:05.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:05.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:30:05.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:06.508 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:30:06.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:06.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:06.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:30:06.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:06.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:30:07.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:08.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:08.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:08.465 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:30:08.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:30:08.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:08.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:30:09.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:10.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:10.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:10.479 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:30:10.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:30:10.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:10.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:30:11.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:12.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:12.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:12.345 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:30:12.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:30:12.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:12.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:30:13.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:14.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:14.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:14.262 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:30:14.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:30:14.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:14.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:30:15.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:16.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:16.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:16.012 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:30:16.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:30:16.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:16.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:30:16.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:17.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:17.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:17.341 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:30:17.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:30:17.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:17.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:30:17.679 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3570710383,v1:127.0.0.1:6819/3570710383] [v2:127.0.0.1:6820/3570710383,v1:127.0.0.1:6821/3570710383] exists,up aec348e5-2271-4288-b27e-ceb84f9d1513 2024-09-13T14:30:17.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:30:17.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:30:17.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:30:17.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-13T14:30:17.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-09-13T14:30:17.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:30:17.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:30:17.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-13T14:30:17.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:30:17.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:30:17.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:30:17.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:30:17.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:30:17.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:30:17.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:30:17.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:30:17.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:30:17.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:30:17.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:30:17.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:30:17.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:30:17.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:30:17.684 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:30:17.684 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:30:17.684 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:30:17.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:30:17.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:30:17.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:30:17.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:30:17.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:30:17.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:30:17.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:30:17.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:30:17.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:30:17.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:30:17.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:30:17.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:30:17.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:30:17.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:30:17.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:30:17.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=626d578c-8ff0-4f1c-87e6-a5a00215d5ff 2024-09-13T14:30:17.686 INFO:tasks.workunit.client.0.smithi112.stdout:add osd3 626d578c-8ff0-4f1c-87e6-a5a00215d5ff 2024-09-13T14:30:17.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 626d578c-8ff0-4f1c-87e6-a5a00215d5ff' 2024-09-13T14:30:17.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:30:17.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB5TORm3b9bKhAAjOIW77cTbES7cfpDXUQ3ew== 2024-09-13T14:30:17.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB5TORm3b9bKhAAjOIW77cTbES7cfpDXUQ3ew=="}' 2024-09-13T14:30:17.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 626d578c-8ff0-4f1c-87e6-a5a00215d5ff -i td/osd-scrub-repair/3/new.json 2024-09-13T14:30:18.185 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:30:18.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-13T14:30:18.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5TORm3b9bKhAAjOIW77cTbES7cfpDXUQ3ew== --osd-uuid 626d578c-8ff0-4f1c-87e6-a5a00215d5ff 2024-09-13T14:30:18.245 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:30:18.248+0000 7f24b08fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:30:18.269 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:30:18.272+0000 7f24b08fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:30:18.271 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:30:18.272+0000 7f24b08fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:30:18.271 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:30:18.272+0000 7f24b08fb7c0 -1 bdev(0x562c374b3c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-13T14:30:18.272 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:30:18.272+0000 7f24b08fb7c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-13T14:30:25.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-13T14:30:25.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:30:25.370 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd3 key to auth repository 2024-09-13T14:30:25.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-13T14:30:25.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:30:25.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-13T14:30:25.769 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.3 2024-09-13T14:30:25.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:30:25.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:30:25.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:30:25.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:30:25.801 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:30:25.800+0000 7fc3d11c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:30:25.803 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:30:25.804+0000 7fc3d11c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:30:25.804 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:30:25.804+0000 7fc3d11c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:30:26.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-13T14:30:26.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:30:26.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:30:26.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:30:26.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:30:26.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:26.114 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:30:26.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:30:26.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:26.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:26.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:27.308 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:30:27.308+0000 7fc3d11c77c0 -1 Falling back to public interface 2024-09-13T14:30:27.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:27.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:27.635 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:30:27.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:30:27.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:27.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:27.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:28.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:28.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:28.949 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:30:28.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:30:28.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:28.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:29.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:30.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:30.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:30.269 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:30:30.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:30:30.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:30.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:30.320 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:30:30.320+0000 7fc3d11c77c0 -1 osd.3 0 log_to_monitors true 2024-09-13T14:30:30.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:31.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:31.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:31.686 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:30:31.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:30:31.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:31.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:32.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:33.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:33.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:33.021 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:30:33.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:30:33.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:33.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:33.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:34.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:34.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:34.360 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:30:34.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:30:34.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:34.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:34.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:35.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:35.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:35.718 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:30:35.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:30:35.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:35.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:36.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:37.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:37.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:37.107 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:30:37.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:30:37.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:37.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:37.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:38.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:38.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:38.513 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:30:38.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:30:38.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:38.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:38.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:39.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:39.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:39.932 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:30:39.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:30:39.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:39.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:40.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:41.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:41.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:41.269 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:30:41.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:30:41.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:41.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:41.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:42.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:42.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:42.610 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:30:42.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:30:42.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:42.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:42.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:43.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:43.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:43.934 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:30:43.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:30:43.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:43.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:44.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:45.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:45.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:45.476 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:30:45.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:30:45.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:45.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:46.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:47.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:47.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:47.202 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:30:47.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:30:47.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:47.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:48.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:49.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:49.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:49.028 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:30:49.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:30:49.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:49.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:49.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:50.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:50.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:50.743 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:30:50.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:30:50.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:50.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:51.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:52.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:52.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:52.758 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:30:52.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:30:52.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:52.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:53.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:54.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:54.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:54.354 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:30:54.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:30:54.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:54.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:55.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:30:56.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:30:56.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:30:56.120 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T14:30:56.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T14:30:56.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:30:56.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:30:56.471 INFO:tasks.workunit.client.0.smithi112.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1960080865,v1:127.0.0.1:6827/1960080865] [v2:127.0.0.1:6828/1960080865,v1:127.0.0.1:6829/1960080865] exists,up 626d578c-8ff0-4f1c-87e6-a5a00215d5ff 2024-09-13T14:30:56.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:30:56.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:30:56.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:30:56.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-09-13T14:30:56.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T14:30:56.786 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' does not exist 2024-09-13T14:30:56.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T14:30:56.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T14:30:57.273 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T14:30:57.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:30:58.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T14:31:01.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-09-13T14:31:01.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:31:01.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:31:01.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:31:01.596 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:31:01.597 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:31:01.597 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:31:01.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:31:01.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:31:01.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:31:01.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:31:01.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:31:01.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:31:01.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:31:01.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:31:01.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:31:02.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:31:02.058 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:31:02.059 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:31:02.059 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T14:31:02.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:31:02.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:02.059 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:31:02.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836508 2024-09-13T14:31:02.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836508 2024-09-13T14:31:02.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508' 2024-09-13T14:31:02.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:02.208 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:31:02.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672983 2024-09-13T14:31:02.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672983 2024-09-13T14:31:02.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672983' 2024-09-13T14:31:02.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:02.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:31:02.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509456 2024-09-13T14:31:02.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509456 2024-09-13T14:31:02.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672983 2-64424509456' 2024-09-13T14:31:02.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:02.494 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:31:02.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345925 2024-09-13T14:31:02.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345925 2024-09-13T14:31:02.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672983 2-64424509456 3-85899345925' 2024-09-13T14:31:02.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:02.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836508 2024-09-13T14:31:02.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:02.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:31:02.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836508 2024-09-13T14:31:02.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:02.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836508 2024-09-13T14:31:02.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-13T14:31:02.662 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836508 2024-09-13T14:31:02.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:31:02.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-13T14:31:02.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:02.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672983 2024-09-13T14:31:02.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:02.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:31:02.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672983 2024-09-13T14:31:02.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:02.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672983 2024-09-13T14:31:02.984 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672983 2024-09-13T14:31:02.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-13T14:31:02.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:31:03.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672982 -lt 42949672983 2024-09-13T14:31:03.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:31:04.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:31:04.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:31:04.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-13T14:31:04.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:04.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509456 2024-09-13T14:31:04.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:04.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:31:04.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509456 2024-09-13T14:31:04.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:04.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509456 2024-09-13T14:31:04.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-13T14:31:04.645 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509456 2024-09-13T14:31:04.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:31:04.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-13T14:31:04.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:04.947 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345925 2024-09-13T14:31:04.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:04.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:31:04.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345925 2024-09-13T14:31:04.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:04.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345925 2024-09-13T14:31:04.953 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.3 seq 85899345925 2024-09-13T14:31:04.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345925' 2024-09-13T14:31:04.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:31:05.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345925 -lt 85899345925 2024-09-13T14:31:05.287 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:31:05.287 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:31:05.287 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:31:05.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T14:31:05.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:31:05.704 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:31:05.704 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:31:05.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:31:05.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:31:05.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:31:05.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:31:06.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:31:06.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:31:06.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:31:06.042 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:31:06.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T14:31:06.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:31:06.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:31:06.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-09-13T14:31:06.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-13T14:31:06.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-13T14:31:06.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-13T14:31:06.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-13T14:31:06.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-13T14:31:06.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-13T14:31:06.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-13T14:31:07.591 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'ecpool' created 2024-09-13T14:31:07.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:31:08.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-13T14:31:08.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-13T14:31:08.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:31:08.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:31:08.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:31:08.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:31:08.618 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:31:08.618 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:31:08.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:31:08.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:31:08.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:31:08.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:31:08.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:31:08.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:31:08.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:31:08.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:31:08.747 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:31:09.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:31:09.111 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:31:09.111 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:31:09.111 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T14:31:09.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:31:09.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:09.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:31:09.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836511 2024-09-13T14:31:09.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836511 2024-09-13T14:31:09.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836511' 2024-09-13T14:31:09.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:09.279 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:31:09.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672985 2024-09-13T14:31:09.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672985 2024-09-13T14:31:09.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836511 1-42949672985' 2024-09-13T14:31:09.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:09.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:31:09.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509458 2024-09-13T14:31:09.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509458 2024-09-13T14:31:09.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836511 1-42949672985 2-64424509458' 2024-09-13T14:31:09.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:09.562 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:31:09.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345927 2024-09-13T14:31:09.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345927 2024-09-13T14:31:09.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836511 1-42949672985 2-64424509458 3-85899345927' 2024-09-13T14:31:09.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:09.706 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836511 2024-09-13T14:31:09.706 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:09.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:31:09.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836511 2024-09-13T14:31:09.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:09.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836511 2024-09-13T14:31:09.709 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836511 2024-09-13T14:31:09.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-09-13T14:31:09.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:31:10.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836511 2024-09-13T14:31:10.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:31:11.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:31:11.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:31:11.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836511 2024-09-13T14:31:11.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:11.359 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672985 2024-09-13T14:31:11.359 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:11.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:31:11.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672985 2024-09-13T14:31:11.363 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:11.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672985 2024-09-13T14:31:11.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-13T14:31:11.364 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672985 2024-09-13T14:31:11.364 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:31:11.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-13T14:31:11.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:11.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509458 2024-09-13T14:31:11.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:11.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:31:11.717 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509458 2024-09-13T14:31:11.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:11.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509458 2024-09-13T14:31:11.719 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509458 2024-09-13T14:31:11.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-13T14:31:11.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:31:12.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-13T14:31:12.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:12.061 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345927 2024-09-13T14:31:12.062 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:12.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:31:12.064 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345927 2024-09-13T14:31:12.065 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:12.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345927 2024-09-13T14:31:12.066 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.3 seq 85899345927 2024-09-13T14:31:12.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2024-09-13T14:31:12.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:31:12.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345928 -lt 85899345927 2024-09-13T14:31:12.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:31:12.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:31:12.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:31:12.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:31:12.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:31:12.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:31:12.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:31:12.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:31:12.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:31:12.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:31:12.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:31:13.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:31:13.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:31:13.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:31:13.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:31:13.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:31:13.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:31:13.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:31:13.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-13T14:31:13.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-13T14:31:13.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-13T14:31:13.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-13T14:31:13.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-13T14:31:13.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T14:31:13.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T14:31:13.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-13T14:31:13.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T14:31:13.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T14:31:13.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T14:31:13.947 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T14:31:13.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T14:31:14.483 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T14:31:14.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T14:31:14.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T14:31:14.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-13T14:31:14.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-13T14:31:14.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-13T14:31:14.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:31:14.850 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:31:14.850 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:31:15.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-13T14:31:15.192 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-13T14:31:15.193 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-13T14:31:15.193 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-13T14:31:15.193 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-13T14:31:15.193 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:31:15.193 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T14:31:15.536 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-13T14:31:15.536 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:31:15.536 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:31:15.537 INFO:tasks.workunit.client.0.smithi112.stderr:0' 2024-09-13T14:31:15.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 0 2024-09-13T14:31:15.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-09-13T14:31:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-13T14:31:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-09-13T14:31:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-09-13T14:31:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-13T14:31:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-13T14:31:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-13T14:31:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-13T14:31:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:31:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:31:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:31:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:31:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:31:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:31:15.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:31:15.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:31:15.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:31:15.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:31:15.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:31:15.540 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:31:15.540 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:31:15.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:31:15.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:31:15.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:31:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:31:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:31:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:31:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:31:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:31:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:31:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:31:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-13T14:31:18.675 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-13T14:31:19.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:31:19.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:31:19.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:31:19.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:31:19.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:31:19.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:31:19.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:31:19.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:31:19.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:31:19.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:31:19.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:31:19.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:31:19.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:31:19.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:31:19.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:31:19.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:31:19.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:31:19.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:31:19.217 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:31:19.218 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:31:19.218 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:31:19.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:31:19.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:31:19.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:31:19.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:31:19.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:31:19.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:31:19.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:31:19.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:31:19.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:31:19.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:31:19.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:31:19.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:31:19.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:31:19.219 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.3 2024-09-13T14:31:19.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:31:19.219 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:31:19.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:31:19.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:31:19.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:31:19.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:31:19.253 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:31:19.250+0000 7fc4608087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:31:19.255 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:31:19.254+0000 7fc4608087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:31:19.257 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:31:19.254+0000 7fc4608087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:31:19.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:31:19.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:31:19.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:31:19.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:31:19.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:31:19.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:31:19.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:31:19.550 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:31:19.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:31:19.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:31:19.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:31:20.720 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:31:20.718+0000 7fc4608087c0 -1 Falling back to public interface 2024-09-13T14:31:20.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:31:20.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:31:20.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:31:20.903 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:31:20.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:31:20.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:31:21.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:31:22.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:31:22.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:31:22.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:31:22.244 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:31:22.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:31:22.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:31:22.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:31:23.515 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:31:23.514+0000 7fc4608087c0 -1 osd.3 38 log_to_monitors true 2024-09-13T14:31:23.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:31:23.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:31:23.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:31:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:31:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:31:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:31:23.968 INFO:tasks.workunit.client.0.smithi112.stderr:osd.3 up in weight 1 up_from 43 up_thru 34 down_at 39 last_clean_interval [20,38) [v2:127.0.0.1:6826/3111555064,v1:127.0.0.1:6827/3111555064] [v2:127.0.0.1:6828/3111555064,v1:127.0.0.1:6829/3111555064] exists,up 626d578c-8ff0-4f1c-87e6-a5a00215d5ff 2024-09-13T14:31:23.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:31:23.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:31:23.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:31:23.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:31:23.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:31:23.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:31:23.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:31:23.969 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:31:23.970 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:31:23.970 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:31:23.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:31:23.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:31:23.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:31:24.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:31:24.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:31:24.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:31:24.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:31:24.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:31:24.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:31:24.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:31:24.573 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:31:24.573 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:31:24.573 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T14:31:24.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:31:24.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:24.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:31:24.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836515 2024-09-13T14:31:24.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836515 2024-09-13T14:31:24.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515' 2024-09-13T14:31:24.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:24.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:31:24.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672989 2024-09-13T14:31:24.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672989 2024-09-13T14:31:24.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-42949672989' 2024-09-13T14:31:24.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:24.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:31:25.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509462 2024-09-13T14:31:25.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509462 2024-09-13T14:31:25.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-42949672989 2-64424509462' 2024-09-13T14:31:25.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:25.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:31:25.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=184683593730 2024-09-13T14:31:25.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 184683593730 2024-09-13T14:31:25.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-42949672989 2-64424509462 3-184683593730' 2024-09-13T14:31:25.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:25.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836515 2024-09-13T14:31:25.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:25.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:31:25.420 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836515 2024-09-13T14:31:25.420 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:25.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836515 2024-09-13T14:31:25.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-09-13T14:31:25.421 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836515 2024-09-13T14:31:25.421 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:31:25.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836514 -lt 21474836515 2024-09-13T14:31:25.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:31:26.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:31:26.752 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:31:27.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836515 -lt 21474836515 2024-09-13T14:31:27.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:27.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:27.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672989 2024-09-13T14:31:27.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:31:27.081 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672989 2024-09-13T14:31:27.081 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:27.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672989 2024-09-13T14:31:27.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-09-13T14:31:27.083 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672989 2024-09-13T14:31:27.083 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:31:27.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672990 -lt 42949672989 2024-09-13T14:31:27.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:27.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509462 2024-09-13T14:31:27.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:27.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:31:27.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509462 2024-09-13T14:31:27.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:27.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509462 2024-09-13T14:31:27.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-09-13T14:31:27.408 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509462 2024-09-13T14:31:27.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:31:27.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509462 -lt 64424509462 2024-09-13T14:31:27.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:27.729 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-184683593730 2024-09-13T14:31:27.729 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:27.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:31:27.731 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-184683593730 2024-09-13T14:31:27.731 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:27.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=184683593730 2024-09-13T14:31:27.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2024-09-13T14:31:27.732 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 184683593730 2024-09-13T14:31:27.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:31:28.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 184683593730 -lt 184683593730 2024-09-13T14:31:28.062 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:31:28.062 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:31:28.062 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:31:28.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:31:28.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:31:28.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:31:28.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:31:28.479 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:31:28.479 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:31:28.479 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:31:28.479 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:31:28.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:31:28.831 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:31:28.831 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:31:28.831 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:31:29.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-13T14:31:29.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-13T14:31:29.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T14:31:29.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-13T14:31:29.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:31:29.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:31:29.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:31:29.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:31:29.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:31:29.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:31:29.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:31:29.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:31:29.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:31:29.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:31:29.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:31:29.629 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:31:29.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:31:29.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:31:30.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-13T14:31:30.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-13T14:31:30.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T14:31:30.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T14:31:30.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T14:31:30.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T14:31:30.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T14:31:30.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T14:31:30.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T14:31:30.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T14:31:30.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T14:31:30.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T14:31:30.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:31:30.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T14:31:30.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:31:30.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:31:30.647 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:31:30.647 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:31:30.647 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:31:30.647 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:31:30.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:31:30.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:31:30.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:31:30.961 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:31:30.961 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:31:30.961 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:31:31.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:31:31.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:31:31.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:31:31.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-13T14:31:31.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:31:31.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:31:31.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:31:31.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:31:31.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-13T14:31:31.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-13T14:31:31.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:31:31.689 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:31:31.689 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:31:31.690 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:31:31.690 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:31:31.690 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:31:32.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:31:07.459736+0000 2024-09-13T14:31:32.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:31:32.301 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:31:32.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:31:07.459736+0000 2024-09-13T14:31:32.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:31:32.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:31:07.459736+0000 2024-09-13T14:31:32.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:31:32.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:31:32.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:31:32.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:31:32.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:31:32.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:31:32.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:31:32.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:31:32.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:31:07.459736+0000 '>' 2024-09-13T14:31:07.459736+0000 2024-09-13T14:31:32.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:31:33.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:31:33.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:31:33.628 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:31:33.628 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:31:33.628 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:31:33.629 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:31:33.629 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:31:33.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:31:07.459736+0000 '>' 2024-09-13T14:31:07.459736+0000 2024-09-13T14:31:33.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:31:34.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:31:34.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:31:34.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:31:34.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:31:34.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:31:34.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:31:34.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:31:35.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:31:32.471834+0000 '>' 2024-09-13T14:31:07.459736+0000 2024-09-13T14:31:35.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:31:35.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:31:37.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:31:37.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:31:37.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:31:37.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:31:37.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:31:37.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:31:37.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:31:37.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:31:37.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:31:37.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:31:37.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:31:37.283 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:31:37.283 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:31:37.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:31:37.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:31:37.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:31:37.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:31:37.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:31:37.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:31:37.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:31:37.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:31:37.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:31:37.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:31:37.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-13T14:31:38.208 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-13T14:31:38.208 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:31:38.208 INFO:tasks.workunit.client.0.smithi112.stdout:hinfo_key 2024-09-13T14:31:38.208 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:31:38.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:31:38.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:31:38.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:31:38.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:31:38.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:31:38.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:31:38.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:31:38.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:31:38.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:31:38.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:31:38.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:31:38.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:31:38.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:31:38.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:31:38.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:31:38.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:31:38.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:31:38.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:31:38.524 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:31:38.524 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:31:38.524 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:31:38.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:31:38.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:31:38.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:31:38.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:31:38.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:31:38.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:31:38.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:31:38.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:31:38.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:31:38.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:31:38.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:31:38.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:31:38.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:31:38.525 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.3 2024-09-13T14:31:38.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:31:38.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:31:38.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:31:38.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:31:38.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:31:38.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:31:38.560 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:31:38.557+0000 7f2754e2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:31:38.651 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:31:38.649+0000 7f2754e2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:31:38.652 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:31:38.649+0000 7f2754e2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:31:38.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:31:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:31:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:31:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:31:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:31:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:31:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:31:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:31:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:31:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:31:39.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:31:39.928 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:31:39.925+0000 7f2754e2f7c0 -1 Falling back to public interface 2024-09-13T14:31:40.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:31:40.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:31:40.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:31:40.265 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:31:40.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:31:40.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:31:40.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:31:41.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:31:41.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:31:41.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:31:41.586 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:31:41.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:31:41.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:31:41.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:31:42.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:31:42.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:31:42.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:31:42.922 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:31:42.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:31:42.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:31:43.236 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:31:43.233+0000 7f2754e2f7c0 -1 osd.3 45 log_to_monitors true 2024-09-13T14:31:43.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:31:44.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:31:44.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:31:44.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:31:44.240 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:31:44.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:31:44.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:31:44.673 INFO:tasks.workunit.client.0.smithi112.stderr:osd.3 up in weight 1 up_from 50 up_thru 50 down_at 46 last_clean_interval [43,45) [v2:127.0.0.1:6826/3323750971,v1:127.0.0.1:6827/3323750971] [v2:127.0.0.1:6828/3323750971,v1:127.0.0.1:6829/3323750971] exists,up 626d578c-8ff0-4f1c-87e6-a5a00215d5ff 2024-09-13T14:31:44.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:31:44.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:31:44.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:31:44.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:31:44.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:31:44.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:31:44.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:31:44.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:31:44.675 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:31:44.675 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:31:44.675 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:31:44.675 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:31:44.675 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:31:44.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:31:44.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:31:44.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:31:44.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:31:44.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:31:44.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:31:45.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:31:45.156 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:31:45.156 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:31:45.156 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T14:31:45.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:31:45.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:45.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:31:45.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836520 2024-09-13T14:31:45.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836520 2024-09-13T14:31:45.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520' 2024-09-13T14:31:45.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:45.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:31:45.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672994 2024-09-13T14:31:45.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672994 2024-09-13T14:31:45.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520 1-42949672994' 2024-09-13T14:31:45.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:45.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:31:45.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509467 2024-09-13T14:31:45.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509467 2024-09-13T14:31:45.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520 1-42949672994 2-64424509467' 2024-09-13T14:31:45.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:45.581 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:31:45.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364802 2024-09-13T14:31:45.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364802 2024-09-13T14:31:45.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520 1-42949672994 2-64424509467 3-214748364802' 2024-09-13T14:31:45.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:45.714 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836520 2024-09-13T14:31:45.714 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:45.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:31:45.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836520 2024-09-13T14:31:45.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:45.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836520 2024-09-13T14:31:45.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-09-13T14:31:45.717 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836520 2024-09-13T14:31:45.717 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:31:46.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836519 -lt 21474836520 2024-09-13T14:31:46.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:31:47.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:31:47.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:31:47.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836520 -lt 21474836520 2024-09-13T14:31:47.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:47.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672994 2024-09-13T14:31:47.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:47.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:31:47.335 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672994 2024-09-13T14:31:47.335 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:47.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672994 2024-09-13T14:31:47.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2024-09-13T14:31:47.336 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672994 2024-09-13T14:31:47.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:31:47.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672995 -lt 42949672994 2024-09-13T14:31:47.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:47.676 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509467 2024-09-13T14:31:47.677 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:47.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:31:47.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509467 2024-09-13T14:31:47.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:47.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509467 2024-09-13T14:31:47.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-09-13T14:31:47.681 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509467 2024-09-13T14:31:47.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:31:48.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509467 -lt 64424509467 2024-09-13T14:31:48.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:48.016 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:48.016 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-214748364802 2024-09-13T14:31:48.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:31:48.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-214748364802 2024-09-13T14:31:48.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:48.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364802 2024-09-13T14:31:48.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 214748364802' 2024-09-13T14:31:48.017 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 214748364802 2024-09-13T14:31:48.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:31:48.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364802 -lt 214748364802 2024-09-13T14:31:48.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:31:48.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:31:48.370 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:31:48.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:31:48.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:31:48.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:31:48.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:31:48.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:31:48.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:31:48.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:31:48.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:31:49.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:31:49.065 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:31:49.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:31:49.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:31:49.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:31:49.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:31:49.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:31:49.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:31:49.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:31:49.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-09-13T14:31:49.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-13T14:31:49.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-13T14:31:49.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-09-13T14:31:49.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:31:49.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:31:49.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:31:49.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:31:49.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:31:49.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:31:49.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:31:49.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:31:49.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:31:49.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:31:49.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:31:49.471 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:31:49.471 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:31:49.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:31:49.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:31:49.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:31:49.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:31:49.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:31:49.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:31:49.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:31:49.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:31:49.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:31:49.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:31:49.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-13T14:31:51.856 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-13T14:31:52.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T14:31:52.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:31:52.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:31:52.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:31:52.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:31:52.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:31:52.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:31:52.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:31:52.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:31:52.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:31:52.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:31:52.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:31:52.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:31:52.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:31:52.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:31:52.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:31:52.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:31:52.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:31:52.397 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:31:52.397 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:31:52.397 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:31:52.397 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:31:52.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:31:52.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:31:52.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:31:52.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:31:52.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:31:52.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:31:52.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:31:52.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:31:52.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:31:52.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:31:52.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:31:52.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:31:52.398 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T14:31:52.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:31:52.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:31:52.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:31:52.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:31:52.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:31:52.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:31:52.432 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:31:52.429+0000 7f1cace427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:31:52.435 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:31:52.433+0000 7f1cace427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:31:52.437 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:31:52.433+0000 7f1cace427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:31:52.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:31:52.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:31:52.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:31:52.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:31:52.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:31:52.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:31:52.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:31:52.738 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:31:52.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:31:52.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:31:53.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:31:53.769 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:31:53.765+0000 7f1cace427c0 -1 Falling back to public interface 2024-09-13T14:31:54.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:31:54.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:31:54.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:31:54.093 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:31:54.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:31:54.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:31:54.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:31:55.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:31:55.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:31:55.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:31:55.424 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:31:55.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:31:55.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:31:55.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:31:56.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:31:56.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:31:56.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:31:56.742 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:31:56.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:31:56.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:31:56.865 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:31:56.861+0000 7f1cace427c0 -1 osd.1 52 log_to_monitors true 2024-09-13T14:31:57.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:31:58.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:31:58.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:31:58.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:31:58.090 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:31:58.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:31:58.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:31:58.611 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [10,52) [v2:127.0.0.1:6810/904626323,v1:127.0.0.1:6811/904626323] [v2:127.0.0.1:6812/904626323,v1:127.0.0.1:6813/904626323] exists,up 4b91e96b-6165-48ee-b271-78e6091f8314 2024-09-13T14:31:58.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:31:58.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:31:58.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:31:58.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:31:58.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:31:58.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:31:58.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:31:58.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:31:58.613 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:31:58.613 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:31:58.613 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:31:58.613 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:31:58.613 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:31:58.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:31:58.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:31:58.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:31:58.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:31:58.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:31:58.732 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:31:59.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:31:59.310 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:31:59.310 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:31:59.310 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T14:31:59.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:31:59.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:59.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:31:59.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836524 2024-09-13T14:31:59.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836524 2024-09-13T14:31:59.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524' 2024-09-13T14:31:59.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:59.498 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:31:59.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135874 2024-09-13T14:31:59.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135874 2024-09-13T14:31:59.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-244813135874' 2024-09-13T14:31:59.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:59.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:31:59.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509471 2024-09-13T14:31:59.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509471 2024-09-13T14:31:59.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-244813135874 2-64424509471' 2024-09-13T14:31:59.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:31:59.789 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:31:59.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364806 2024-09-13T14:31:59.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364806 2024-09-13T14:31:59.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-244813135874 2-64424509471 3-214748364806' 2024-09-13T14:31:59.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:31:59.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836524 2024-09-13T14:31:59.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:31:59.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:31:59.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:31:59.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836524 2024-09-13T14:31:59.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836524 2024-09-13T14:31:59.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-13T14:31:59.936 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836524 2024-09-13T14:31:59.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:32:00.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836522 -lt 21474836524 2024-09-13T14:32:00.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:32:01.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:32:01.263 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:32:01.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-13T14:32:01.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:01.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135874 2024-09-13T14:32:01.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:01.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:32:01.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135874 2024-09-13T14:32:01.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:01.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135874 2024-09-13T14:32:01.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-09-13T14:32:01.589 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 244813135874 2024-09-13T14:32:01.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:32:01.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135874 -lt 244813135874 2024-09-13T14:32:01.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:01.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509471 2024-09-13T14:32:01.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:01.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:32:01.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509471 2024-09-13T14:32:01.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:01.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509471 2024-09-13T14:32:01.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-09-13T14:32:01.936 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509471 2024-09-13T14:32:01.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:32:02.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509471 -lt 64424509471 2024-09-13T14:32:02.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:02.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-214748364806 2024-09-13T14:32:02.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:02.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:32:02.275 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-214748364806 2024-09-13T14:32:02.275 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:02.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364806 2024-09-13T14:32:02.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 214748364806' 2024-09-13T14:32:02.277 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 214748364806 2024-09-13T14:32:02.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:32:02.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364806 -lt 214748364806 2024-09-13T14:32:02.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:32:02.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:32:02.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:32:02.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:32:02.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:32:02.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:32:02.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:32:02.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:32:02.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:32:02.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:32:02.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:32:03.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:32:03.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:32:03.311 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:32:03.311 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:32:03.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:32:03.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:32:03.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:32:03.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-13T14:32:03.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:32:03.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:32:03.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:32:03.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:32:04.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-13T14:32:04.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-13T14:32:04.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:32:04.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:32:04.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:32:04.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:32:04.036 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:32:04.036 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:32:04.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:31:32.471834+0000 2024-09-13T14:32:04.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:32:04.643 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:32:04.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:31:32.471834+0000 2024-09-13T14:32:04.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:32:04.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:31:32.471834+0000 2024-09-13T14:32:04.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:32:04.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:32:04.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:32:04.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:32:04.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:32:04.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:32:04.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:32:04.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:32:04.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:31:32.471834+0000 '>' 2024-09-13T14:31:32.471834+0000 2024-09-13T14:32:04.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:32:05.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:32:05.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:32:05.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:32:05.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:32:05.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:32:05.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:32:05.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:32:06.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:31:32.471834+0000 '>' 2024-09-13T14:31:32.471834+0000 2024-09-13T14:32:06.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:32:07.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:32:07.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:32:07.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:32:07.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:32:07.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:32:07.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:32:07.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:32:07.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:31:32.471834+0000 '>' 2024-09-13T14:31:32.471834+0000 2024-09-13T14:32:07.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:32:08.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:32:08.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:32:08.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:32:08.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:32:08.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:32:08.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:32:08.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:32:08.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:31:32.471834+0000 '>' 2024-09-13T14:31:32.471834+0000 2024-09-13T14:32:08.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:32:09.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:32:09.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:32:09.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:32:09.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:32:09.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:32:09.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:32:09.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:32:10.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:32:05.265190+0000 '>' 2024-09-13T14:31:32.471834+0000 2024-09-13T14:32:10.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:32:10.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:32:12.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:32:12.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:32:12.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:32:12.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:32:12.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:32:12.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:32:12.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:32:12.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:32:12.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:32:12.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:32:12.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:32:12.283 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:32:12.283 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:32:12.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:32:12.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:32:12.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:32:12.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:32:12.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:32:12.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:32:12.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:32:12.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:32:12.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:32:12.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:32:12.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-13T14:32:13.186 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-13T14:32:13.186 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:32:13.187 INFO:tasks.workunit.client.0.smithi112.stdout:hinfo_key 2024-09-13T14:32:13.187 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:32:13.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:32:13.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:32:13.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:32:13.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:32:13.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:32:13.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:32:13.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:32:13.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:32:13.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:32:13.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:32:13.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:32:13.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:32:13.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:32:13.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:32:13.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:32:13.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:32:13.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:32:13.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:32:13.481 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:32:13.481 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:32:13.482 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:32:13.482 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:32:13.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:32:13.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:32:13.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:32:13.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:32:13.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:32:13.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:32:13.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:32:13.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:32:13.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:32:13.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:32:13.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:32:13.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:32:13.484 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.3 2024-09-13T14:32:13.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:32:13.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:32:13.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:32:13.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:32:13.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:32:13.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:32:13.517 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:32:13.516+0000 7f44158777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:32:13.619 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:32:13.616+0000 7f44158777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:32:13.620 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:32:13.616+0000 7f44158777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:32:13.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:32:13.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:32:13.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:32:13.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:32:13.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:32:13.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:13.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:32:13.873 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:32:13.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:13.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:32:14.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:14.932 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:32:14.928+0000 7f44158777c0 -1 Falling back to public interface 2024-09-13T14:32:15.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:15.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:15.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:32:15.217 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:32:15.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:15.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:32:15.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:16.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:16.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:16.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:32:16.554 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:32:16.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:16.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:32:16.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:17.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:17.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:17.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:32:17.937 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:32:17.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:17.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:32:18.035 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:32:18.032+0000 7f44158777c0 -1 osd.3 59 log_to_monitors true 2024-09-13T14:32:18.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:19.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:19.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:19.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:32:19.274 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:32:19.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:19.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:32:19.764 INFO:tasks.workunit.client.0.smithi112.stderr:osd.3 up in weight 1 up_from 64 up_thru 64 down_at 60 last_clean_interval [50,59) [v2:127.0.0.1:6826/1436453619,v1:127.0.0.1:6827/1436453619] [v2:127.0.0.1:6828/1436453619,v1:127.0.0.1:6829/1436453619] exists,up 626d578c-8ff0-4f1c-87e6-a5a00215d5ff 2024-09-13T14:32:19.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:32:19.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:32:19.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:32:19.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:32:19.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:32:19.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:32:19.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:32:19.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:32:19.765 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:32:19.765 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:32:19.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:32:19.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:32:19.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:32:19.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:32:19.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:32:19.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:32:19.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:32:19.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:32:19.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:32:20.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:32:20.234 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:32:20.234 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:32:20.234 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T14:32:20.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:32:20.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:20.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:32:20.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836529 2024-09-13T14:32:20.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836529 2024-09-13T14:32:20.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529' 2024-09-13T14:32:20.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:20.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:32:20.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135879 2024-09-13T14:32:20.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135879 2024-09-13T14:32:20.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529 1-244813135879' 2024-09-13T14:32:20.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:20.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:32:20.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509476 2024-09-13T14:32:20.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509476 2024-09-13T14:32:20.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529 1-244813135879 2-64424509476' 2024-09-13T14:32:20.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:20.671 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:32:20.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906946 2024-09-13T14:32:20.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906946 2024-09-13T14:32:20.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836529 1-244813135879 2-64424509476 3-274877906946' 2024-09-13T14:32:20.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:20.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836529 2024-09-13T14:32:20.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:20.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:32:20.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836529 2024-09-13T14:32:20.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:20.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836529 2024-09-13T14:32:20.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-09-13T14:32:20.815 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836529 2024-09-13T14:32:20.816 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:32:21.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836528 -lt 21474836529 2024-09-13T14:32:21.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:32:22.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:32:22.121 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:32:22.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836529 -lt 21474836529 2024-09-13T14:32:22.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:22.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135879 2024-09-13T14:32:22.466 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:22.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:32:22.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:22.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135879 2024-09-13T14:32:22.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135879 2024-09-13T14:32:22.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135879' 2024-09-13T14:32:22.469 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 244813135879 2024-09-13T14:32:22.469 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:32:22.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135880 -lt 244813135879 2024-09-13T14:32:22.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:22.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509476 2024-09-13T14:32:22.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:22.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:32:22.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509476 2024-09-13T14:32:22.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:22.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509476 2024-09-13T14:32:22.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2024-09-13T14:32:22.809 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509476 2024-09-13T14:32:22.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:32:23.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509477 -lt 64424509476 2024-09-13T14:32:23.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:23.150 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906946 2024-09-13T14:32:23.151 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:23.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:32:23.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:23.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906946 2024-09-13T14:32:23.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906946 2024-09-13T14:32:23.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906946' 2024-09-13T14:32:23.154 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 274877906946 2024-09-13T14:32:23.154 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:32:23.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906946 -lt 274877906946 2024-09-13T14:32:23.508 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:32:23.508 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:32:23.508 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:32:23.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:32:23.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:32:23.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:32:23.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:32:23.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:32:23.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:32:23.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:32:23.950 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:32:24.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:32:24.289 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:32:24.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:32:24.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:32:24.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:32:24.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:32:24.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:32:24.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:32:24.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:32:24.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-09-13T14:32:24.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-13T14:32:24.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-13T14:32:24.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-09-13T14:32:24.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-09-13T14:32:24.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-13T14:32:24.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:32:24.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:32:24.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:32:24.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 158493"' 2024-09-13T14:32:24.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 158493' 2024-09-13T14:32:24.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T14:32:24.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:32:24.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:32:24.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:32:24.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 158496"' 2024-09-13T14:32:24.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/158495: /' 2024-09-13T14:32:24.731 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 158496' 2024-09-13T14:32:24.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-13T14:32:24.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 158493 158496' 2024-09-13T14:32:24.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:32:24.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:32:24.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 158493 2024-09-13T14:32:24.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T14:32:24.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/158499: /' 2024-09-13T14:32:28.139 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:32:28.140 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:32:28.140 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:32:28.140 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:32:28.140 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:32:28.140 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:32:28.140 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:32:28.140 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:32:28.140 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:32:28.140 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:32:28.140 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:32:28.140 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:32:28.140 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:32:28.141 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:32:28.141 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:32:28.141 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:32:28.141 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:32:28.141 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:32:28.141 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:32:28.141 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:32:28.141 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:32:28.141 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:32:28.141 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-13T14:32:28.141 INFO:tasks.workunit.client.0.smithi112.stderr:158495: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-13T14:32:28.141 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T14:32:28.141 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:32:28.142 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:32:28.142 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:32:28.142 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:32:28.142 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:32:28.142 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:32:28.142 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:32:28.145 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:32:28.145 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:32:28.145 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:32:28.145 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:32:28.145 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:32:28.145 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:32:28.145 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:32:28.145 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:32:28.146 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:32:28.146 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:32:28.146 INFO:tasks.workunit.client.0.smithi112.stderr:158495: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:32:28.146 INFO:tasks.workunit.client.0.smithi112.stderr:158495: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:32:28.146 INFO:tasks.workunit.client.0.smithi112.stderr:158495: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:32:28.146 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:32:28.146 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:32:28.146 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:32:28.146 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:32:28.146 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:32:28.146 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:32:28.146 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:32:28.146 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:32:28.147 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:32:28.147 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:32:28.147 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:32:28.147 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:32:28.147 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:32:28.147 INFO:tasks.workunit.client.0.smithi112.stderr:158495: start osd.1 2024-09-13T14:32:28.147 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:32:28.221 INFO:tasks.workunit.client.0.smithi112.stderr:1158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:32:28.221 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:32:28.221 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-13T14:32:28.222 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:32:28.222 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T14:32:28.222 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:32:28.222 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:32:28.222 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-13T14:32:28.222 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:32:28.222 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-13T14:32:28.222 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:32:28.222 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:32:28.222 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:32:28.222 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:32:28.222 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:32:28.222 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:32:28.223 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T14:32:28.223 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:32:28.223 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:32:28.223 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-13T14:32:28.223 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:32:28.223 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:32:28.223 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-13T14:32:28.223 INFO:tasks.workunit.client.0.smithi112.stderr:158499: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-13T14:32:28.223 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-13T14:32:28.223 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:32:28.224 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:32:28.224 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T14:32:28.224 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:32:28.224 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:32:28.224 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:32:28.224 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:32:28.224 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:32:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:32:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T14:32:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T14:32:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:32:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:32:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:32:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:32:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:32:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:32:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:158499: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:32:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:158499: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:32:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:158499: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:32:28.226 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:32:28.226 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:32:28.226 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:32:28.226 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:32:28.226 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:32:28.226 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:32:28.226 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:32:28.226 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:32:28.226 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:32:28.226 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:32:28.226 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:32:28.226 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T14:32:28.226 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T14:32:28.227 INFO:tasks.workunit.client.0.smithi112.stderr:158499: start osd.2 2024-09-13T14:32:28.227 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:32:32.875 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:32:32.876 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:32:32.876 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:32:32.876 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:32:32.876 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:32:32.876 INFO:tasks.workunit.client.0.smithi112.stderr:158495: 2024-09-13T14:32:28.184+0000 7f9dd74427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:32:32.876 INFO:tasks.workunit.client.0.smithi112.stderr:158495: 2024-09-13T14:32:28.188+0000 7f9dd74427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:32:32.876 INFO:tasks.workunit.client.0.smithi112.stderr:158495: 2024-09-13T14:32:28.188+0000 7f9dd74427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:32:32.876 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:32:32.876 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:32:32.876 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:32:32.876 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:32:32.877 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:32:32.877 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:32.877 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:32:32.877 INFO:tasks.workunit.client.0.smithi112.stderr:158495: 0 2024-09-13T14:32:32.877 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:32.877 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:32:32.877 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:32.877 INFO:tasks.workunit.client.0.smithi112.stderr:158495: 2024-09-13T14:32:29.780+0000 7f9dd74427c0 -1 Falling back to public interface 2024-09-13T14:32:32.877 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:32.877 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:32.877 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:32:32.877 INFO:tasks.workunit.client.0.smithi112.stderr:158495: 1 2024-09-13T14:32:32.878 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:32.878 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:32:32.878 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:32.878 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:32.878 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:32.878 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:32:32.878 INFO:tasks.workunit.client.0.smithi112.stderr:158495: 2 2024-09-13T14:32:32.878 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:32.878 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:32:32.878 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:32.878 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:32.878 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:32.878 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:32:32.879 INFO:tasks.workunit.client.0.smithi112.stderr:158495: 3 2024-09-13T14:32:32.879 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:32.879 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:32:32.879 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:33.490 INFO:tasks.workunit.client.0.smithi112.stderr:158495: 2024-09-13T14:32:32.876+0000 7f9dd74427c0 -58499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-13T14:32:33.491 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T14:32:33.491 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:32:33.491 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:32:33.491 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:32:33.491 INFO:tasks.workunit.client.0.smithi112.stderr:158499: 2024-09-13T14:32:28.252+0000 7f9e6e57d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:32:33.491 INFO:tasks.workunit.client.0.smithi112.stderr:158499: 2024-09-13T14:32:28.256+0000 7f9e6e57d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:32:33.491 INFO:tasks.workunit.client.0.smithi112.stderr:158499: 2024-09-13T14:32:28.256+0000 7f9e6e57d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:32:33.491 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T14:32:33.491 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:32:33.491 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:32:33.491 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:32:33.491 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:32:33.491 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:33.492 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:32:33.492 INFO:tasks.workunit.client.0.smithi112.stderr:158499: 0 2024-09-13T14:32:33.492 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:33.492 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:32:33.492 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:33.492 INFO:tasks.workunit.client.0.smithi112.stderr:158499: 2024-09-13T14:32:29.848+0000 7f9e6e57d7c0 -1 Falling back to public interface 2024-09-13T14:32:33.492 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:33.492 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:33.492 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:32:33.492 INFO:tasks.workunit.client.0.smithi112.stderr:158499: 1 2024-09-13T14:32:33.492 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:33.492 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:32:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:32:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:158499: 2 2024-09-13T14:32:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:32:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:32:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:158499: 3 2024-09-13T14:32:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:33.493 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:32:33.494 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:34.932 INFO:tasks.workunit.client.0.smithi112.stderr:158499: 2024-09-13T14:32:33.492+0000 7f9e6e57d7c0 -1 osd.1 65 log_to_monitors true 2024-09-13T14:32:34.932 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:34.932 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:34.932 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:32:34.932 INFO:tasks.workunit.client.0.smithi112.stderr:158495: 4 2024-09-13T14:32:34.932 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:34.932 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:32:34.932 INFO:tasks.workunit.client.0.smithi112.stderr:158495: osd.1 up in weight 1 up_from 70 up_thru 60 down_at 66 last_clean_interval [57,65) [v2:127.0.0.1:6810/1634378285,v1:127.0.0.1:6811/1634378285] [v2:127.0.0.1:6812/1634378285,v1:127.0.0.1:6813/1634378285] exists,up 4b91e96b-6165-48ee-b271-78e6091f8314 2024-09-13T14:32:34.932 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:32:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:32:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:32:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:32:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:32:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:32:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:32:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:32:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:158495: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:32:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:158495: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:32:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:32:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:32:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:32:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:32:34.933 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:32:34.934 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:32:34.934 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:32:34.934 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:32:34.934 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:32:34.934 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:32:34.934 INFO:tasks.workunit.client.0.smithi112.stderr:158495: 1 2024-09-13T14:32:34.934 INFO:tasks.workunit.client.0.smithi112.stderr:158495: 2 2024-09-13T14:32:34.934 INFO:tasks.workunit.client.0.smithi112.stderr:158495: 3' 2024-09-13T14:32:34.934 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:32:34.934 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:34.934 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:32:34.934 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836533 2024-09-13T14:32:34.934 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836533 2024-09-13T14:32:34.934 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533' 2024-09-13T14:32:34.935 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:36.125 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-help1 osd.2 65 log_to_monitors true 2024-09-13T14:32:36.125 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:36.125 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:36.126 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:32:36.126 INFO:tasks.workunit.client.0.smithi112.stderr:158499: 4 2024-09-13T14:32:36.126 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:36.126 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:32:36.126 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:36.126 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:36.126 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:36.126 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:32:36.126 INFO:tasks.workunit.client.0.smithi112.stderr:158499: 5 2024-09-13T14:32:36.126 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:36.126 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:32:36.126 INFO:tasks.workunit.client.0.smithi112.stderr:158499: osd.2 up in weight 1 up_from 73 up_thru 53 down_at 66 last_clean_interval [15,65) [v2:127.0.0.1:6818/2741165877,v1:127.0.0.1:6819/2741165877] [v2:127.0.0.1:6820/2741165877,v1:127.0.0.1:6821/2741165877] exists,up aec348e5-2271-4288-b27e-ceb84f9d1513 2024-09-13T14:32:36.126 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:32:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:32:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:32:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:32:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:32:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:32:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:32:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:32:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:158499: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:32:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:158499: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:32:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:32:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:32:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:32:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:32:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:32:36.127 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:32:36.128 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:32:36.128 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:32:36.128 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:32:36.128 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:32:36.128 INFO:tasks.workunit.client.0.smithi112.stderr:158499: 1 2024-09-13T14:32:36.128 INFO:tasks.workunit.client.0.smithi112.stderr:158499: 2 2024-09-13T14:32:36.128 INFO:tasks.workunit.client.0.smithi112.stderr:158499: 3' 2024-09-13T14:32:36.128 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:32:36.128 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:37.443 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/uers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:32:37.443 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=300647710722 2024-09-13T14:32:37.443 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 300647710722 2024-09-13T14:32:37.443 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-300647710722' 2024-09-13T14:32:37.443 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:37.443 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:32:37.443 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612610 2024-09-13T14:32:37.443 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612610 2024-09-13T14:32:37.443 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-300647710722 2-313532612610' 2024-09-13T14:32:37.444 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:37.444 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:32:37.444 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906950 2024-09-13T14:32:37.444 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906950 2024-09-13T14:32:37.444 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-300647710722 2-313532612610 3-274877906950' 2024-09-13T14:32:37.444 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:37.444 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836533 2024-09-13T14:32:37.444 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:37.444 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:32:37.444 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836533 2024-09-13T14:32:37.444 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:37.444 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836533 2024-09-13T14:32:37.445 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-09-13T14:32:37.445 INFO:tasks.workunit.client.0.smithi112.stderr:158495: waiting osd.0 seq 21474836533 2024-09-13T14:32:37.445 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:32:37.445 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836532 -lt 21474836533 2024-09-13T14:32:37.445 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:32:37.445 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:32:37.445 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:32:37.445 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836533 -lt 21474836533 2024-09-13T14:32:37.445 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:37.445 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-300647710722 2024-09-13T14:32:37.445 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:37.445 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:32:37.446 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-300647710722 2024-09-13T14:32:37.446 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:38.404 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubbuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:32:38.404 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836534 2024-09-13T14:32:38.404 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836534 2024-09-13T14:32:38.404 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836534' 2024-09-13T14:32:38.404 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:38.404 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:32:38.404 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=300647710723 2024-09-13T14:32:38.404 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 300647710723 2024-09-13T14:32:38.404 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836534 1-300647710723' 2024-09-13T14:32:38.404 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:38.405 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:32:38.405 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612611 2024-09-13T14:32:38.405 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612611 2024-09-13T14:32:38.405 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836534 1-300647710723 2-313532612611' 2024-09-13T14:32:38.405 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:38.405 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:32:38.405 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906951 2024-09-13T14:32:38.405 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906951 2024-09-13T14:32:38.405 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836534 1-300647710723 2-313532612611 3-274877906951' 2024-09-13T14:32:38.405 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:38.405 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836534 2024-09-13T14:32:38.405 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:38.406 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:32:38.406 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836534 2024-09-13T14:32:38.406 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:38.406 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836534 2024-09-13T14:32:38.406 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-09-13T14:32:38.406 INFO:tasks.workunit.client.0.smithi112.stderr:158499: waiting osd.0 seq 21474836534 2024-09-13T14:32:38.406 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:32:38.406 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836533 -lt 21474836534 2024-09-13T14:32:38.406 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:32:38.406 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:32:38.406 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:32:38.406 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836534 -lt 21474836534 2024-09-13T14:32:38.854 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/untu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=300647710722 2024-09-13T14:32:38.854 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 300647710722' 2024-09-13T14:32:38.854 INFO:tasks.workunit.client.0.smithi112.stderr:158495: waiting osd.1 seq 300647710722 2024-09-13T14:32:38.854 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:32:38.854 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 300647710722 -lt 300647710722 2024-09-13T14:32:38.855 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:38.855 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-313532612610 2024-09-13T14:32:38.855 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:38.855 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:32:38.855 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-313532612610 2024-09-13T14:32:38.855 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:38.855 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612610 2024-09-13T14:32:38.855 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-09-13T14:32:38.855 INFO:tasks.workunit.client.0.smithi112.stderr:158495: waiting osd.2 seq 313532612610 2024-09-13T14:32:38.856 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:32:38.856 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612610 -lt 313532612610 2024-09-13T14:32:38.856 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:38.856 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906950 2024-09-13T14:32:38.856 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:38.856 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:32:38.856 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906950 2024-09-13T14:32:38.856 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:38.856 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906950 2024-09-13T14:32:38.857 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906950' 2024-09-13T14:32:38.857 INFO:tasks.workunit.client.0.smithi112.stderr:158495: waiting osd.3 seq 274877906950 2024-09-13T14:32:38.857 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:32:38.857 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906951 -lt 274877906950 2024-09-13T14:32:38.857 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:32:38.857 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:32:38.857 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:32:38.857 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:32:38.857 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:32:38.857 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:32:38.858 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:32:38.858 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:32:39.574 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:32:39.574 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:32:39.574 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:32:39.574 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:32:39.575 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:32:39.575 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:32:39.575 INFO:tasks.workunit.client.0.smithi112.stderr:158495: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:32:39.575 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:32:39.575 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:32:39.575 INFO:tasks.workunit.client.0.smithi112.stderr:158495: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:32:39.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:32:39.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:32:39.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 158496 2024-09-13T14:32:39.824 INFO:tasks.workunit.client.0.smithi112.stderr:standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:39.824 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-300647710723 2024-09-13T14:32:39.824 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:39.824 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:32:39.824 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-300647710723 2024-09-13T14:32:39.824 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:39.824 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=300647710723 2024-09-13T14:32:39.824 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 300647710723' 2024-09-13T14:32:39.824 INFO:tasks.workunit.client.0.smithi112.stderr:158499: waiting osd.1 seq 300647710723 2024-09-13T14:32:39.824 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:32:39.824 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 300647710723 -lt 300647710723 2024-09-13T14:32:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-313532612611 2024-09-13T14:32:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:32:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-313532612611 2024-09-13T14:32:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612611 2024-09-13T14:32:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 313532612611' 2024-09-13T14:32:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:158499: waiting osd.2 seq 313532612611 2024-09-13T14:32:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:32:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612611 -lt 313532612611 2024-09-13T14:32:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906951 2024-09-13T14:32:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:32:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906951 2024-09-13T14:32:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906951 2024-09-13T14:32:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906951' 2024-09-13T14:32:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:158499: waiting osd.3 seq 274877906951 2024-09-13T14:32:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:32:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906951 -lt 274877906951 2024-09-13T14:32:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:32:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:32:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:32:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:32:40.530 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:32:40.530 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:32:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:32:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:32:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:32:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:32:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:32:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:32:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:32:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:32:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:158499: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:32:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:32:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:32:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:158499: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:32:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:32:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:32:40.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:32:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:32:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-13T14:32:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:32:40.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-13T14:32:40.533 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:32:40.533 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:32:40.533 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:32:40.533 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:32:40.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-13T14:32:40.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-13T14:32:40.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:32:40.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:32:40.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:32:40.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:32:40.878 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:32:40.878 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:32:41.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:32:05.265190+0000 2024-09-13T14:32:41.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:32:41.519 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:32:41.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:32:05.265190+0000 2024-09-13T14:32:41.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:32:41.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:32:05.265190+0000 2024-09-13T14:32:41.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:32:41.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:32:41.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:32:41.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:32:41.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:32:41.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:32:41.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:32:41.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:32:41.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:32:05.265190+0000 '>' 2024-09-13T14:32:05.265190+0000 2024-09-13T14:32:41.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:32:42.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:32:42.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:32:42.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:32:42.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:32:42.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:32:42.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:32:42.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:32:43.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:32:05.265190+0000 '>' 2024-09-13T14:32:05.265190+0000 2024-09-13T14:32:43.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:32:44.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:32:44.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:32:44.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:32:44.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:32:44.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:32:44.210 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:32:44.210 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:32:44.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:32:41.976894+0000 '>' 2024-09-13T14:32:05.265190+0000 2024-09-13T14:32:44.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:32:44.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:32:46.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-13T14:32:46.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:32:46.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:32:46.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:32:46.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 161388"' 2024-09-13T14:32:46.542 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 161388' 2024-09-13T14:32:46.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-13T14:32:46.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:32:46.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:32:46.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:32:46.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/161390: /' 2024-09-13T14:32:46.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 161391"' 2024-09-13T14:32:46.542 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 161391' 2024-09-13T14:32:46.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-13T14:32:46.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 161388 161391' 2024-09-13T14:32:46.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:32:46.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:32:46.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 161388 2024-09-13T14:32:46.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-13T14:32:46.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/161393: /' 2024-09-13T14:32:47.728 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:32:47.728 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:32:47.728 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:32:47.728 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:32:47.729 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:32:47.729 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:32:47.729 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:32:47.729 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:32:47.729 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:32:47.729 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:32:47.729 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:32:47.729 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:32:47.729 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:32:47.729 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:32:47.729 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:32:47.729 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:32:47.730 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:32:47.730 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:32:47.730 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:32:47.730 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:32:47.730 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:32:47.730 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:32:47.730 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-13T14:32:47.730 INFO:tasks.workunit.client.0.smithi112.stderr:161390: _ 2024-09-13T14:32:47.730 INFO:tasks.workunit.client.0.smithi112.stderr:161390: hinfo_key 2024-09-13T14:32:47.730 INFO:tasks.workunit.client.0.smithi112.stderr:161390: snapset 2024-09-13T14:32:47.730 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T14:32:47.730 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:32:47.730 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:32:47.731 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:32:47.731 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:32:47.731 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:32:47.731 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:32:47.731 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:32:47.733 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:32:47.733 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:32:47.733 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:32:47.733 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:32:47.734 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:32:47.734 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:32:47.734 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:32:47.734 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:32:47.734 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:32:47.734 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:32:47.734 INFO:tasks.workunit.client.0.smithi112.stderr:161390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:32:47.734 INFO:tasks.workunit.client.0.smithi112.stderr:161390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:32:47.734 INFO:tasks.workunit.client.0.smithi112.stderr:161390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:32:47.734 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:32:47.734 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:32:47.734 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:32:47.734 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:32:47.735 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:32:47.735 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:32:47.735 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:32:47.735 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:32:47.735 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:32:47.735 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:32:47.735 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:32:47.735 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:32:47.735 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:32:47.735 INFO:tasks.workunit.client.0.smithi112.stderr:161390: start osd.1 2024-09-13T14:32:47.742 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:32:47.742 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:32:47.742 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-13T14:32:47.742 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:32:47.742 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-13T14:32:47.742 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:32:47.742 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:32:47.742 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-13T14:32:47.742 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:32:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-13T14:32:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:32:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:32:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:32:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:32:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:32:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:32:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-13T14:32:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:32:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:32:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-13T14:32:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:32:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:32:47.744 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-09-13T14:32:47.744 INFO:tasks.workunit.client.0.smithi112.stderr:161393: _ 2024-09-13T14:32:47.744 INFO:tasks.workunit.client.0.smithi112.stderr:161393: hinfo_key 2024-09-13T14:32:47.744 INFO:tasks.workunit.client.0.smithi112.stderr:161393: snapset 2024-09-13T14:32:47.744 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-13T14:32:47.744 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:32:47.744 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:32:47.744 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T14:32:47.744 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:32:47.744 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:32:47.744 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:32:47.744 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:32:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:32:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:32:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T14:32:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T14:32:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:32:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:32:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:32:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:32:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:32:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:32:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:161393: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:32:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:161393: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:32:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:161393: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:32:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:32:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:32:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:32:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:32:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:32:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:32:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:32:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:32:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:32:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:32:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:32:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T14:32:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T14:32:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:161393: start osd.2 2024-09-13T14:32:53.285 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:32:53.285 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:32:53.286 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:32:53.286 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:32:53.286 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:32:53.286 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:32:53.286 INFO:tasks.workunit.client.0.smithi112.stderr:161390: 2024-09-13T14:32:47.771+0000 7fd9f8feb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:32:53.286 INFO:tasks.workunit.client.0.smithi112.stderr:161390: 2024-09-13T14:32:47.811+0000 7fd9f8feb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:32:53.286 INFO:tasks.workunit.client.0.smithi112.stderr:161390: 2024-09-13T14:32:47.815+0000 7fd9f8feb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:32:53.286 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:32:53.286 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:32:53.286 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:32:53.286 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:32:53.286 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:32:53.286 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:53.287 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:32:53.287 INFO:tasks.workunit.client.0.smithi112.stderr:161390: 0 2024-09-13T14:32:53.287 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:53.287 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:32:53.287 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:53.287 INFO:tasks.workunit.client.0.smithi112.stderr:161390: 2024-09-13T14:32:49.335+0000 7fd9f8feb7c0 -1 Falling back to public interface 2024-09-13T14:32:53.287 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:53.287 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:53.287 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:32:53.287 INFO:tasks.workunit.client.0.smithi112.stderr:161390: 1 2024-09-13T14:32:53.287 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:53.287 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:32:53.287 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:53.288 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:53.288 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:53.288 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:32:53.288 INFO:tasks.workunit.client.0.smithi112.stderr:161390: 2 2024-09-13T14:32:53.288 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:53.288 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:32:53.288 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:53.288 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:53.288 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:53.288 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:32:53.288 INFO:tasks.workunit.client.0.smithi112.stderr:161390: 3 2024-09-13T14:32:53.288 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:53.288 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:32:53.289 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:53.460 INFO:tasks.workunit.client.0.smithi112.stderr:161390: 2024-09-13T14:32:53.271+0000 7fd9very_ops 2024-09-13T14:32:53.460 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-13T14:32:53.460 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T14:32:53.460 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:32:53.460 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:32:53.460 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:32:53.460 INFO:tasks.workunit.client.0.smithi112.stderr:161393: 2024-09-13T14:32:47.791+0000 7f5a595237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:32:53.460 INFO:tasks.workunit.client.0.smithi112.stderr:161393: 2024-09-13T14:32:47.815+0000 7f5a595237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:32:53.460 INFO:tasks.workunit.client.0.smithi112.stderr:161393: 2024-09-13T14:32:47.815+0000 7f5a595237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:32:53.461 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T14:32:53.461 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:32:53.461 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:32:53.461 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:32:53.461 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:32:53.461 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:53.461 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:32:53.461 INFO:tasks.workunit.client.0.smithi112.stderr:161393: 0 2024-09-13T14:32:53.461 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:53.461 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:32:53.461 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:53.461 INFO:tasks.workunit.client.0.smithi112.stderr:161393: 2024-09-13T14:32:49.407+0000 7f5a595237c0 -1 Falling back to public interface 2024-09-13T14:32:53.461 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:53.461 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:53.462 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:32:53.462 INFO:tasks.workunit.client.0.smithi112.stderr:161393: 1 2024-09-13T14:32:53.462 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:53.462 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:32:53.462 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:53.462 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:53.462 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:53.462 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:32:53.462 INFO:tasks.workunit.client.0.smithi112.stderr:161393: 2 2024-09-13T14:32:53.462 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:32:53.462 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:53.462 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:53.462 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:53.462 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:53.463 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:32:53.463 INFO:tasks.workunit.client.0.smithi112.stderr:161393: 3 2024-09-13T14:32:53.463 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:53.463 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:32:53.463 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:55.776 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:55.777 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:55.777 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:32:55.777 INFO:tasks.workunit.client.0.smithi112.stderr:161393: 4 2024-09-13T14:32:55.777 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:55.777 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:32:55.777 INFO:tasks.workunit.client.0.smithi112.stderr:161393: 2024-09-13T14:32:53.503+0000 7f5a595237c0 -1 osd.2 75 log_to_monitors true 2024-09-13T14:32:55.777 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:55.777 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:55.777 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:55.777 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:32:55.777 INFO:tasks.workunit.client.0.smithi112.stderr:161393: 5 2024-09-13T14:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:161393: osd.2 up in weight 1 up_from 82 up_thru 53 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6818/1791577748,v1:127.0.0.1:6819/1791577748] [v2:127.0.0.1:6820/1791577748,v1:127.0.0.1:6821/1791577748] exists,up aec348e5-2271-4288-b27e-ceb84f9d1513 2024-09-13T14:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:161393: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:161393: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:32:55.779 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:32:55.779 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:32:55.779 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:32:55.779 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:32:55.779 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:32:55.779 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:32:55.779 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:32:55.779 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:32:55.779 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:32:55.779 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:32:55.779 INFO:tasks.workunit.client.0.smithi112.stderr:161393: 1 2024-09-13T14:32:55.779 INFO:tasks.workunit.client.0.smithi112.stderr:161393: 2 2024-09-13T14:32:55.779 INFO:tasks.workunit.client.0.smithi112.stderr:161393: 3' 2024-09-13T14:32:55.780 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:32:55.780 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:55.780 INFO:tasks.workunit.client.0.smithi112.stderr:161393f8feb7c0 -1 osd.1 75 log_to_monitors true 2024-09-13T14:32:55.780 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:55.780 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:55.780 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:32:55.780 INFO:tasks.workunit.client.0.smithi112.stderr:161390: 4 2024-09-13T14:32:55.780 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:55.780 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:32:55.780 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:32:55.780 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:32:55.780 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:32:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:32:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:161390: 5 2024-09-13T14:32:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:32:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:32:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:161390: osd.1 up in weight 1 up_from 81 up_thru 81 down_at 76 last_clean_interval [70,75) [v2:127.0.0.1:6810/4170712802,v1:127.0.0.1:6811/4170712802] [v2:127.0.0.1:6812/4170712802,v1:127.0.0.1:6813/4170712802] exists,up 4b91e96b-6165-48ee-b271-78e6091f8314 2024-09-13T14:32:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:32:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:32:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:32:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:32:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:32:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:32:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:32:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:32:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:161390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:32:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:161390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:32:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:32:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:32:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:32:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:32:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:32:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:32:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:32:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:32:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:32:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:32:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:161390: 1 2024-09-13T14:32:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:161390: 2 2024-09-13T14:32:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:161390: 3' 2024-09-13T14:32:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:32:55.783 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:56.958 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:32:56.958 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836539 2024-09-13T14:32:56.958 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836539 2024-09-13T14:32:56.958 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836539' 2024-09-13T14:32:56.958 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:56.958 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:32:56.958 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350979 2024-09-13T14:32:56.958 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350979 2024-09-13T14:32:56.958 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836539 1-347892350979' 2024-09-13T14:32:56.958 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:56.958 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:32:56.959 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318275 2024-09-13T14:32:56.959 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318275 2024-09-13T14:32:56.959 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836539 1-347892350979 2-352187318275' 2024-09-13T14:32:56.959 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:56.959 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:32:56.959 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906957 2024-09-13T14:32:56.959 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906957 2024-09-13T14:32:56.959 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836539 1-347892350979 2-352187318275 3-274877906957' 2024-09-13T14:32:56.959 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:56.959 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836539 2024-09-13T14:32:56.959 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:56.959 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:32:56.960 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836539 2024-09-13T14:32:56.960 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:56.960 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836539 2024-09-13T14:32:56.960 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2024-09-13T14:32:56.960 INFO:tasks.workunit.client.0.smithi112.stderr:161393: waiting osd.0 seq 21474836539 2024-09-13T14:32:56.960 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:32:56.960 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836540 -lt 21474836539 2024-09-13T14:32:56.960 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:56.960 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-347892350979 2024-09-13T14:32:56.960 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:56.960 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:32:56.961 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-h: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:32:56.961 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836540 2024-09-13T14:32:56.961 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836540 2024-09-13T14:32:56.961 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540' 2024-09-13T14:32:56.961 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:56.961 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:32:56.961 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350978 2024-09-13T14:32:56.961 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350978 2024-09-13T14:32:56.961 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-347892350978' 2024-09-13T14:32:56.961 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:56.961 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:32:56.961 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318274 2024-09-13T14:32:56.962 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318274 2024-09-13T14:32:56.962 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-347892350978 2-352187318274' 2024-09-13T14:32:56.962 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:32:56.962 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:32:56.962 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906956 2024-09-13T14:32:56.962 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906956 2024-09-13T14:32:56.962 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-347892350978 2-352187318274 3-274877906956' 2024-09-13T14:32:56.962 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:56.962 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836540 2024-09-13T14:32:56.963 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:56.963 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:32:56.963 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836540 2024-09-13T14:32:56.963 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:56.963 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836540 2024-09-13T14:32:56.963 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-09-13T14:32:56.963 INFO:tasks.workunit.client.0.smithi112.stderr:161390: waiting osd.0 seq 21474836540 2024-09-13T14:32:56.963 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:32:56.963 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836540 -lt 21474836540 2024-09-13T14:32:56.963 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:56.963 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-347892350978 2024-09-13T14:32:56.963 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:56.963 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:32:59.716 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-347892350979 2024-09-13T14:32:59.716 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:59.716 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350979 2024-09-13T14:32:59.717 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 347892350979' 2024-09-13T14:32:59.717 INFO:tasks.workunit.client.0.smithi112.stderr:161393: waiting osd.1 seq 347892350979 2024-09-13T14:32:59.717 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:32:59.717 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 347892350979 2024-09-13T14:32:59.717 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:32:59.717 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:32:59.717 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:32:59.717 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350979 -lt 347892350979 2024-09-13T14:32:59.717 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:59.717 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318275 2024-09-13T14:32:59.717 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:59.717 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:32:59.717 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318275 2024-09-13T14:32:59.717 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:59.718 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318275 2024-09-13T14:32:59.718 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318275' 2024-09-13T14:32:59.718 INFO:tasks.workunit.client.0.smithi112.stderr:161393: waiting osd.2 seq 352187318275 2024-09-13T14:32:59.718 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:32:59.718 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318275 -lt 352187318275 2024-09-13T14:32:59.718 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:59.718 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906957 2024-09-13T14:32:59.718 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:59.718 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:32:59.718 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906957 2024-09-13T14:32:59.718 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:59.718 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906957 2024-09-13T14:32:59.718 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906957' 2024-09-13T14:32:59.718 INFO:tasks.workunit.client.0.smithi112.stderr:161393: waiting osd.3 seq 274877906957 2024-09-13T14:32:59.719 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:32:59.719 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906957 -lt 274877906957 2024-09-13T14:32:59.719 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:32:59.719 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:32:59.719 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:32:59.719 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:32:59.719 INFO:tasks.workunit.client.0.smithi112.stderr:16elpers.sh:2220: flush_pg_stats: echo 1-347892350978 2024-09-13T14:32:59.719 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:59.719 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350978 2024-09-13T14:32:59.719 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 347892350978' 2024-09-13T14:32:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:161390: waiting osd.1 seq 347892350978 2024-09-13T14:32:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:32:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 347892350978 2024-09-13T14:32:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:32:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:32:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:32:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350979 -lt 347892350978 2024-09-13T14:32:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318274 2024-09-13T14:32:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:32:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318274 2024-09-13T14:32:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318274 2024-09-13T14:32:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318274' 2024-09-13T14:32:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:161390: waiting osd.2 seq 352187318274 2024-09-13T14:32:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:32:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318275 -lt 352187318274 2024-09-13T14:32:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:32:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906956 2024-09-13T14:32:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:32:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:32:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906956 2024-09-13T14:32:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:32:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906956 2024-09-13T14:32:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906956' 2024-09-13T14:32:59.721 INFO:tasks.workunit.client.0.smithi112.stderr:161390: waiting osd.3 seq 274877906956 2024-09-13T14:32:59.722 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:32:59.722 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906957 -lt 274877906956 2024-09-13T14:32:59.722 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:32:59.722 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:32:59.722 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:32:59.722 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:33:01.250 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:33:01.250 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:33:01.250 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:33:01.250 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:33:01.250 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:33:01.250 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:33:01.250 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:33:01.250 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T14:33:01.250 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:33:01.250 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:33:01.250 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:33:01.251 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-13T14:33:01.251 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-13T14:33:01.251 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T14:33:01.251 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-13T14:33:01.251 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:33:01.251 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:33:01.251 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:33:01.251 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:33:01.251 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:33:01.251 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:33:01.251 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:33:01.251 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:33:01.251 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:33:01.251 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:33:01.252 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:33:01.252 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:33:01.252 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:33:01.252 INFO:tasks.workunit.client.0.smithi112.stderr:161390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:33:01.252 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:33:01.252 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:33:01.252 INFO:tasks.workunit.client.0.smithi112.stderr:161390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:33:01.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:33:01.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:33:01.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 161391 2024-09-13T14:33:01.252 INFO:tasks.workunit.client.0.smithi112.stderr:1393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:33:01.252 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:33:01.252 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:33:01.252 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:33:01.253 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:33:01.253 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:33:01.253 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:33:01.253 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T14:33:01.253 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:33:01.253 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:33:01.253 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:33:01.253 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-13T14:33:01.253 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-13T14:33:01.253 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T14:33:01.253 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-13T14:33:01.253 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:33:01.253 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:33:01.253 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:33:01.254 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:33:01.254 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:33:01.254 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:33:01.254 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:33:01.254 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:33:01.254 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:33:01.254 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:33:01.254 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:33:01.254 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:33:01.254 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:33:01.255 INFO:tasks.workunit.client.0.smithi112.stderr:161393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:33:01.255 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:33:01.255 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:33:01.255 INFO:tasks.workunit.client.0.smithi112.stderr:161393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:33:01.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:33:01.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:33:01.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:33:01.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:33:01.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-13T14:33:01.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:33:01.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:33:01.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:33:01.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-09-13T14:33:01.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-13T14:33:01.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-13T14:33:01.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-13T14:33:01.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-09-13T14:33:01.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-13T14:33:01.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:33:01.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:33:01.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:33:01.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 163916"' 2024-09-13T14:33:01.308 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 163916' 2024-09-13T14:33:01.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:33:01.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:33:01.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:33:01.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:33:01.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 163918"' 2024-09-13T14:33:01.308 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 163918' 2024-09-13T14:33:01.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/163919: /' 2024-09-13T14:33:01.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-13T14:33:01.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 163916 163918' 2024-09-13T14:33:01.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:33:01.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:33:01.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 163916 2024-09-13T14:33:01.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:33:01.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/163922: /' 2024-09-13T14:33:04.681 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:33:04.681 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:33:04.681 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:33:04.681 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:33:04.681 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:33:04.681 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:33:04.681 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:33:04.681 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:33:04.681 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:33:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:33:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:33:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:33:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:33:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:33:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:33:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:33:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:33:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:33:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:33:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:33:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:33:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:33:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-13T14:33:04.683 INFO:tasks.workunit.client.0.smithi112.stderr:163919: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-13T14:33:04.683 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:33:04.683 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:33:04.683 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:33:04.683 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:33:04.683 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:33:04.683 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:33:04.683 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:33:04.683 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:33:04.687 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:33:04.687 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:33:04.687 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:33:04.687 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:33:04.687 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:33:04.687 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:33:04.687 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:33:04.688 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:33:04.688 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:33:04.688 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:33:04.688 INFO:tasks.workunit.client.0.smithi112.stderr:163919: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:33:04.688 INFO:tasks.workunit.client.0.smithi112.stderr:163919: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:33:04.688 INFO:tasks.workunit.client.0.smithi112.stderr:163919: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:33:04.688 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:33:04.688 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:33:04.688 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:33:04.688 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:33:04.688 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:33:04.688 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:33:04.688 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:33:04.689 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:33:04.689 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:33:04.689 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:33:04.689 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:33:04.689 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:33:04.689 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:33:04.689 INFO:tasks.workunit.client.0.smithi112.stderr:163919: start osd.3 2024-09-13T14:33:04.689 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:33:05.106 INFO:tasks.workunit.client.0.smithi112.stderr:1163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:33:05.106 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:33:05.106 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:33:05.107 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:33:05.107 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:33:05.107 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:33:05.107 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:33:05.107 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:33:05.107 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:33:05.107 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:33:05.107 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:33:05.107 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:33:05.107 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:33:05.107 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:33:05.107 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:33:05.108 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:33:05.108 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:33:05.108 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:33:05.108 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:33:05.108 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:33:05.108 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:33:05.108 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:33:05.108 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-13T14:33:05.108 INFO:tasks.workunit.client.0.smithi112.stderr:163922: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-13T14:33:05.109 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T14:33:05.109 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:33:05.109 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:33:05.109 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:33:05.109 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:33:05.109 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:33:05.109 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:33:05.109 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:33:05.110 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:33:05.110 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:33:05.110 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:33:05.110 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:33:05.110 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:33:05.110 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:33:05.110 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:33:05.111 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:33:05.111 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:33:05.111 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:33:05.111 INFO:tasks.workunit.client.0.smithi112.stderr:163922: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:33:05.111 INFO:tasks.workunit.client.0.smithi112.stderr:163922: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:33:05.111 INFO:tasks.workunit.client.0.smithi112.stderr:163922: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:33:05.111 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:33:05.111 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:33:05.111 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:33:05.111 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:33:05.111 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:33:05.111 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:33:05.111 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:33:05.111 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:33:05.112 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:33:05.112 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:33:05.112 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:33:05.112 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:33:05.112 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:33:05.112 INFO:tasks.workunit.client.0.smithi112.stderr:163922: start osd.1 2024-09-13T14:33:05.112 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:33:09.477 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:33:09.477 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:33:09.477 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:33:09.477 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:33:09.477 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:33:09.478 INFO:tasks.workunit.client.0.smithi112.stderr:163919: 2024-09-13T14:33:04.719+0000 7fc29c3377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:09.478 INFO:tasks.workunit.client.0.smithi112.stderr:163919: 2024-09-13T14:33:04.767+0000 7fc29c3377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:09.478 INFO:tasks.workunit.client.0.smithi112.stderr:163919: 2024-09-13T14:33:04.767+0000 7fc29c3377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:09.478 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:33:09.478 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:33:09.478 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:33:09.478 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:33:09.478 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:33:09.478 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:09.478 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:33:09.478 INFO:tasks.workunit.client.0.smithi112.stderr:163919: 0 2024-09-13T14:33:09.479 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:09.479 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:33:09.479 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:09.479 INFO:tasks.workunit.client.0.smithi112.stderr:163919: 2024-09-13T14:33:06.135+0000 7fc29c3377c0 -1 Falling back to public interface 2024-09-13T14:33:09.479 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:09.479 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:09.479 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:33:09.479 INFO:tasks.workunit.client.0.smithi112.stderr:163919: 1 2024-09-13T14:33:09.479 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:09.479 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:33:09.479 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:09.479 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:09.479 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:09.479 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:33:09.480 INFO:tasks.workunit.client.0.smithi112.stderr:163919: 2 2024-09-13T14:33:09.480 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:09.480 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:33:09.480 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:09.480 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:09.480 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:09.480 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:33:09.480 INFO:tasks.workunit.client.0.smithi112.stderr:163919: 3 2024-09-13T14:33:09.480 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:09.480 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:33:09.480 INFO:tasks.workunit.client.0.smithi112.stderr:163919: 2024-09-13T14:33:09.115+0000 7fc29c3377c0 -1 osd.3 85 log_to_monitors true 2024-09-13T14:33:10.323 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers63922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:33:10.323 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:33:10.323 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:33:10.324 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:33:10.324 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:33:10.324 INFO:tasks.workunit.client.0.smithi112.stderr:163922: 2024-09-13T14:33:05.139+0000 7f32905b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:10.324 INFO:tasks.workunit.client.0.smithi112.stderr:163922: 2024-09-13T14:33:05.151+0000 7f32905b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:10.324 INFO:tasks.workunit.client.0.smithi112.stderr:163922: 2024-09-13T14:33:05.151+0000 7f32905b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:10.324 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:33:10.324 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:33:10.324 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:33:10.324 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:33:10.324 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:33:10.324 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:10.324 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:33:10.324 INFO:tasks.workunit.client.0.smithi112.stderr:163922: 0 2024-09-13T14:33:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:33:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:163922: 2024-09-13T14:33:06.647+0000 7f32905b37c0 -1 Falling back to public interface 2024-09-13T14:33:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:33:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:163922: 1 2024-09-13T14:33:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:33:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:10.326 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:33:10.326 INFO:tasks.workunit.client.0.smithi112.stderr:163922: 2 2024-09-13T14:33:10.326 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:10.326 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:33:10.326 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:10.326 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:10.326 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:10.326 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:33:10.326 INFO:tasks.workunit.client.0.smithi112.stderr:163922: 3 2024-09-13T14:33:10.326 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:10.326 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:33:10.326 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:11.655 INFO:tasks.workunit.client.0.smithi112.stderr:163922: 2024-09-13T14:33:10.323+0000 7f32905b37c0 -.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:11.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:11.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:11.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:33:11.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: 4 2024-09-13T14:33:11.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:11.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:33:11.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: osd.3 up in weight 1 up_from 90 up_thru 90 down_at 86 last_clean_interval [64,85) [v2:127.0.0.1:6810/3854134575,v1:127.0.0.1:6811/3854134575] [v2:127.0.0.1:6812/3854134575,v1:127.0.0.1:6813/3854134575] exists,up 626d578c-8ff0-4f1c-87e6-a5a00215d5ff 2024-09-13T14:33:11.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:33:11.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:33:11.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:33:11.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:33:11.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:33:11.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:33:11.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:33:11.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:33:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:163919: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:33:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:163919: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:33:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:33:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:33:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:33:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:33:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:33:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:33:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:33:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:33:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:33:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:33:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:163919: 1 2024-09-13T14:33:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:163919: 2 2024-09-13T14:33:11.658 INFO:tasks.workunit.client.0.smithi112.stderr:163919: 3' 2024-09-13T14:33:11.658 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:33:11.658 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:11.658 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:33:11.658 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836544 2024-09-13T14:33:11.658 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836544 2024-09-13T14:33:11.658 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836544' 2024-09-13T14:33:11.658 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:12.210 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-help1 osd.1 85 log_to_monitors true 2024-09-13T14:33:12.210 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:12.210 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:12.210 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:33:12.210 INFO:tasks.workunit.client.0.smithi112.stderr:163922: 4 2024-09-13T14:33:12.210 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:12.210 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:33:12.211 INFO:tasks.workunit.client.0.smithi112.stderr:163922: osd.1 up in weight 1 up_from 93 up_thru 82 down_at 86 last_clean_interval [81,85) [v2:127.0.0.1:6826/3848671997,v1:127.0.0.1:6827/3848671997] [v2:127.0.0.1:6828/3848671997,v1:127.0.0.1:6829/3848671997] exists,up 4b91e96b-6165-48ee-b271-78e6091f8314 2024-09-13T14:33:12.211 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:33:12.211 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:33:12.211 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:33:12.211 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:33:12.211 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:33:12.211 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:33:12.211 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:33:12.211 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:33:12.211 INFO:tasks.workunit.client.0.smithi112.stderr:163922: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:33:12.211 INFO:tasks.workunit.client.0.smithi112.stderr:163922: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:33:12.211 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:33:12.211 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:33:12.211 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:33:12.212 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:33:12.212 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:33:12.212 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:33:12.212 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:33:12.212 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:33:12.212 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:33:12.212 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:33:12.212 INFO:tasks.workunit.client.0.smithi112.stderr:163922: 1 2024-09-13T14:33:12.212 INFO:tasks.workunit.client.0.smithi112.stderr:163922: 2 2024-09-13T14:33:12.212 INFO:tasks.workunit.client.0.smithi112.stderr:163922: 3' 2024-09-13T14:33:12.212 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:33:12.212 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:12.212 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:33:12.213 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836545 2024-09-13T14:33:12.213 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836545 2024-09-13T14:33:12.213 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545' 2024-09-13T14:33:12.213 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:13.212 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:33:13.212 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=399431958530 2024-09-13T14:33:13.212 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 399431958530 2024-09-13T14:33:13.212 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836544 1-399431958530' 2024-09-13T14:33:13.212 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:13.212 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:33:13.212 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318279 2024-09-13T14:33:13.212 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318279 2024-09-13T14:33:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836544 1-399431958530 2-352187318279' 2024-09-13T14:33:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:33:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=386547056642 2024-09-13T14:33:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 386547056642 2024-09-13T14:33:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836544 1-399431958530 2-352187318279 3-386547056642' 2024-09-13T14:33:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:33:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836544 2024-09-13T14:33:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:33:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:33:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836544 2024-09-13T14:33:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:33:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836544 2024-09-13T14:33:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-09-13T14:33:13.214 INFO:tasks.workunit.client.0.smithi112.stderr:163919: waiting osd.0 seq 21474836544 2024-09-13T14:33:13.214 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:33:13.214 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836545 -lt 21474836544 2024-09-13T14:33:13.214 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:33:13.214 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-399431958530 2024-09-13T14:33:13.214 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:33:13.214 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:33:13.214 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-399431958530 2024-09-13T14:33:13.214 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:33:13.214 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=399431958530 2024-09-13T14:33:13.214 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 399431958530' 2024-09-13T14:33:13.214 INFO:tasks.workunit.client.0.smithi112.stderr:163919: waiting osd.1 seq 399431958530 2024-09-13T14:33:13.214 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:33:13.493 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222ers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:33:13.493 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=399431958531 2024-09-13T14:33:13.493 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 399431958531 2024-09-13T14:33:13.493 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545 1-399431958531' 2024-09-13T14:33:13.494 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:13.494 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:33:13.494 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318280 2024-09-13T14:33:13.494 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318280 2024-09-13T14:33:13.494 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545 1-399431958531 2-352187318280' 2024-09-13T14:33:13.494 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:13.494 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:33:13.494 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=386547056643 2024-09-13T14:33:13.494 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 386547056643 2024-09-13T14:33:13.494 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545 1-399431958531 2-352187318280 3-386547056643' 2024-09-13T14:33:13.494 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:33:13.494 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836545 2024-09-13T14:33:13.494 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:33:13.494 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:33:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836545 2024-09-13T14:33:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:33:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836545 2024-09-13T14:33:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836545' 2024-09-13T14:33:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:163922: waiting osd.0 seq 21474836545 2024-09-13T14:33:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:33:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836545 -lt 21474836545 2024-09-13T14:33:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:33:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-399431958531 2024-09-13T14:33:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:33:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:33:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-399431958531 2024-09-13T14:33:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:33:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=399431958531 2024-09-13T14:33:13.495 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 399431958531' 2024-09-13T14:33:13.496 INFO:tasks.workunit.client.0.smithi112.stderr:163922: waiting osd.1 seq 399431958531 2024-09-13T14:33:13.496 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:33:15.653 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 399431958530 -lt 399431958530 2024-09-13T14:33:15.653 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:33:15.653 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318279 2024-09-13T14:33:15.654 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:33:15.654 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:33:15.654 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318279 2024-09-13T14:33:15.654 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:33:15.654 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318279 2024-09-13T14:33:15.654 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318279' 2024-09-13T14:33:15.654 INFO:tasks.workunit.client.0.smithi112.stderr:163919: waiting osd.2 seq 352187318279 2024-09-13T14:33:15.654 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:33:15.654 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318279 -lt 352187318279 2024-09-13T14:33:15.654 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:33:15.654 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-386547056642 2024-09-13T14:33:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:33:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:33:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-386547056642 2024-09-13T14:33:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:33:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=386547056642 2024-09-13T14:33:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 386547056642' 2024-09-13T14:33:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: waiting osd.3 seq 386547056642 2024-09-13T14:33:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:33:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 386547056642 2024-09-13T14:33:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:33:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:33:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:33:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 386547056643 -lt 386547056642 2024-09-13T14:33:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:33:15.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:33:15.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:33:15.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:33:15.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:33:15.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:33:15.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:33:15.656 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:33:15.991 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | : flush_pg_stats: test 399431958530 -lt 399431958531 2024-09-13T14:33:15.991 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:33:15.991 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:33:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:33:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 399431958531 -lt 399431958531 2024-09-13T14:33:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:33:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318280 2024-09-13T14:33:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:33:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:33:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318280 2024-09-13T14:33:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:33:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318280 2024-09-13T14:33:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318280' 2024-09-13T14:33:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:163922: waiting osd.2 seq 352187318280 2024-09-13T14:33:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:33:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318281 -lt 352187318280 2024-09-13T14:33:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:33:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-386547056643 2024-09-13T14:33:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:33:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:33:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-386547056643 2024-09-13T14:33:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:33:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=386547056643 2024-09-13T14:33:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 386547056643' 2024-09-13T14:33:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:163922: waiting osd.3 seq 386547056643 2024-09-13T14:33:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:33:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 386547056643 -lt 386547056643 2024-09-13T14:33:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:33:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:33:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:33:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:33:15.994 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:33:15.994 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:33:15.994 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:33:15.994 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:33:16.741 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:33:16.741 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:33:16.742 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:33:16.742 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:33:16.742 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:33:16.742 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:33:16.742 INFO:tasks.workunit.client.0.smithi112.stderr:163922: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:33:16.742 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:33:16.742 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:33:16.742 INFO:tasks.workunit.client.0.smithi112.stderr:163922: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:33:16.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:33:17.222 INFO:tasks.workunit.client.0.smithi112.stderr:not)' 2024-09-13T14:33:17.222 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:33:17.223 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:33:17.223 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T14:33:17.223 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:33:17.223 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:33:17.223 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:33:17.223 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-13T14:33:17.223 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-13T14:33:17.223 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T14:33:17.223 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-13T14:33:17.223 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:33:17.223 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:33:17.223 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:33:17.224 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:33:17.224 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:33:17.224 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:33:17.224 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:33:17.224 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:33:17.224 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:33:17.224 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:33:17.224 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:33:17.224 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:33:17.224 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:33:17.224 INFO:tasks.workunit.client.0.smithi112.stderr:163919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:33:17.224 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:33:17.224 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:33:17.225 INFO:tasks.workunit.client.0.smithi112.stderr:163919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:33:17.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:33:17.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:33:17.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 163918 2024-09-13T14:33:17.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:33:17.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:33:17.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:33:17.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-13T14:33:17.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:33:17.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-13T14:33:17.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:33:17.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:33:17.226 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:33:17.226 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:33:17.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-13T14:33:17.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-13T14:33:17.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:33:17.548 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:33:17.548 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:33:17.548 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:33:17.548 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:33:17.548 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:33:17.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:32:41.976894+0000 2024-09-13T14:33:17.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:33:18.192 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:33:18.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:32:41.976894+0000 2024-09-13T14:33:18.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:33:18.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:32:41.976894+0000 2024-09-13T14:33:18.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:33:18.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:33:18.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:33:18.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:33:18.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:33:18.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:33:18.210 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:33:18.210 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:33:18.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:32:41.976894+0000 '>' 2024-09-13T14:32:41.976894+0000 2024-09-13T14:33:18.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:33:19.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:33:19.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:33:19.506 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:33:19.506 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:33:19.506 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:33:19.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:33:19.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:33:19.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:32:41.976894+0000 '>' 2024-09-13T14:32:41.976894+0000 2024-09-13T14:33:19.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:33:20.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:33:20.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:33:20.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:33:20.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:33:20.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:33:20.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:33:20.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:33:21.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:33:19.077653+0000 '>' 2024-09-13T14:32:41.976894+0000 2024-09-13T14:33:21.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:33:21.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:33:23.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-13T14:33:23.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:33:23.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:33:23.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:33:23.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 166837"' 2024-09-13T14:33:23.174 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 166837' 2024-09-13T14:33:23.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:33:23.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:33:23.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:33:23.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:33:23.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/166839: /' 2024-09-13T14:33:23.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 166840"' 2024-09-13T14:33:23.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 166840' 2024-09-13T14:33:23.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-13T14:33:23.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 166837 166840' 2024-09-13T14:33:23.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:33:23.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:33:23.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 166837 2024-09-13T14:33:23.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:33:23.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/166843: /' 2024-09-13T14:33:24.388 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:33:24.388 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:33:24.388 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:33:24.388 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:33:24.388 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:33:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:33:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:33:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:33:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:33:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:33:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:33:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:33:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:33:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:33:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:33:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:33:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:33:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:33:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:33:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:33:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:33:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:33:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-13T14:33:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:166839: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-13T14:33:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:166839: _ 2024-09-13T14:33:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:166839: hinfo_key 2024-09-13T14:33:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:166839: snapset 2024-09-13T14:33:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:33:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:33:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:33:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:33:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:33:24.391 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:33:24.391 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:33:24.394 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:33:24.394 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:33:24.394 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:33:24.395 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:33:24.395 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:33:24.395 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:33:24.395 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:33:24.395 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:33:24.395 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:33:24.395 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:33:24.395 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:33:24.395 INFO:tasks.workunit.client.0.smithi112.stderr:166839: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:33:24.395 INFO:tasks.workunit.client.0.smithi112.stderr:166839: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:33:24.395 INFO:tasks.workunit.client.0.smithi112.stderr:166839: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:33:24.395 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:33:24.395 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:33:24.395 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:33:24.396 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:33:24.396 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:33:24.396 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:33:24.396 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:33:24.396 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:33:24.396 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:33:24.396 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:33:24.396 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:33:24.396 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:33:24.396 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:33:24.396 INFO:tasks.workunit.client.0.smithi112.stderr:166839: start osd.3 2024-09-13T14:33:24.472 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-l166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:33:24.472 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:33:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:33:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:33:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:33:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:33:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:33:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:33:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:33:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:33:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:33:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:33:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:33:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:33:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:33:24.474 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:33:24.474 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:33:24.474 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:33:24.474 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:33:24.474 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:33:24.474 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:33:24.474 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:33:24.475 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-13T14:33:24.475 INFO:tasks.workunit.client.0.smithi112.stderr:166843: _ 2024-09-13T14:33:24.475 INFO:tasks.workunit.client.0.smithi112.stderr:166843: hinfo_key 2024-09-13T14:33:24.475 INFO:tasks.workunit.client.0.smithi112.stderr:166843: snapset 2024-09-13T14:33:24.475 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T14:33:24.475 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:33:24.475 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:33:24.475 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:33:24.475 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:33:24.475 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:33:24.475 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:33:24.475 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:33:24.476 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:33:24.476 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:33:24.476 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:33:24.476 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:33:24.476 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:33:24.476 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:33:24.476 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:33:24.476 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:33:24.477 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:33:24.477 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:33:24.477 INFO:tasks.workunit.client.0.smithi112.stderr:166843: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:33:24.477 INFO:tasks.workunit.client.0.smithi112.stderr:166843: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:33:24.477 INFO:tasks.workunit.client.0.smithi112.stderr:166843: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:33:24.477 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:33:24.477 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:33:24.477 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:33:24.477 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:33:24.477 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:33:24.477 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:33:24.477 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:33:24.478 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:33:24.478 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:33:24.478 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:33:24.478 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:33:24.478 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:33:24.478 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:33:24.478 INFO:tasks.workunit.client.0.smithi112.stderr:166843: start osd.1 2024-09-13T14:33:29.064 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recoen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:33:29.064 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:33:29.065 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:33:29.065 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:33:29.065 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:33:29.065 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:33:29.065 INFO:tasks.workunit.client.0.smithi112.stderr:166839: 2024-09-13T14:33:24.426+0000 7f56368fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:29.065 INFO:tasks.workunit.client.0.smithi112.stderr:166839: 2024-09-13T14:33:24.434+0000 7f56368fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:29.065 INFO:tasks.workunit.client.0.smithi112.stderr:166839: 2024-09-13T14:33:24.434+0000 7f56368fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:29.065 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:33:29.065 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:33:29.066 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:33:29.066 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:33:29.066 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:33:29.066 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:29.066 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:33:29.066 INFO:tasks.workunit.client.0.smithi112.stderr:166839: 0 2024-09-13T14:33:29.066 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:29.066 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:33:29.066 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:29.066 INFO:tasks.workunit.client.0.smithi112.stderr:166839: 2024-09-13T14:33:25.974+0000 7f56368fb7c0 -1 Falling back to public interface 2024-09-13T14:33:29.066 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:29.066 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:29.066 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:33:29.066 INFO:tasks.workunit.client.0.smithi112.stderr:166839: 1 2024-09-13T14:33:29.067 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:29.067 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:33:29.067 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:29.067 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:29.067 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:29.067 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:33:29.067 INFO:tasks.workunit.client.0.smithi112.stderr:166839: 2 2024-09-13T14:33:29.067 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:29.067 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:33:29.067 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:29.067 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:29.067 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:29.067 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:33:29.067 INFO:tasks.workunit.client.0.smithi112.stderr:166839: 3 2024-09-13T14:33:29.068 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:29.068 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:33:30.179 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.very_ops 2024-09-13T14:33:30.179 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:33:30.179 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:33:30.179 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:33:30.179 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:33:30.179 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:33:30.179 INFO:tasks.workunit.client.0.smithi112.stderr:166843: 2024-09-13T14:33:24.490+0000 7f1d2940c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:30.179 INFO:tasks.workunit.client.0.smithi112.stderr:166843: 2024-09-13T14:33:24.494+0000 7f1d2940c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:30.179 INFO:tasks.workunit.client.0.smithi112.stderr:166843: 2024-09-13T14:33:24.498+0000 7f1d2940c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:30.179 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:33:30.179 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:33:30.180 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:33:30.180 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:33:30.180 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:33:30.180 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:30.180 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:33:30.180 INFO:tasks.workunit.client.0.smithi112.stderr:166843: 0 2024-09-13T14:33:30.180 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:30.180 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:33:30.180 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:30.180 INFO:tasks.workunit.client.0.smithi112.stderr:166843: 2024-09-13T14:33:26.042+0000 7f1d2940c7c0 -1 Falling back to public interface 2024-09-13T14:33:30.180 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:30.180 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:30.180 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:33:30.181 INFO:tasks.workunit.client.0.smithi112.stderr:166843: 1 2024-09-13T14:33:30.181 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:30.181 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:33:30.181 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:30.181 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:30.181 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:30.181 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:33:30.181 INFO:tasks.workunit.client.0.smithi112.stderr:166843: 2 2024-09-13T14:33:30.181 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:30.181 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:33:30.181 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:30.181 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:30.181 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:30.182 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:33:30.182 INFO:tasks.workunit.client.0.smithi112.stderr:166843: 3 2024-09-13T14:33:30.182 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:30.182 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:33:30.182 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:31.446 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.clien0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:31.446 INFO:tasks.workunit.client.0.smithi112.stderr:166839: 2024-09-13T14:33:29.690+0000 7f56368fb7c0 -1 osd.3 95 log_to_monitors true 2024-09-13T14:33:31.446 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:31.446 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:31.446 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:33:31.446 INFO:tasks.workunit.client.0.smithi112.stderr:166839: 4 2024-09-13T14:33:31.446 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:31.446 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:33:31.446 INFO:tasks.workunit.client.0.smithi112.stderr:166839: osd.3 up in weight 1 up_from 99 up_thru 93 down_at 96 last_clean_interval [90,95) [v2:127.0.0.1:6810/2614252688,v1:127.0.0.1:6811/2614252688] [v2:127.0.0.1:6812/2614252688,v1:127.0.0.1:6813/2614252688] exists,up 626d578c-8ff0-4f1c-87e6-a5a00215d5ff 2024-09-13T14:33:31.446 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:33:31.447 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:33:31.447 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:33:31.447 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:33:31.447 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:33:31.447 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:33:31.447 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:33:31.447 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:33:31.447 INFO:tasks.workunit.client.0.smithi112.stderr:166839: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:33:31.447 INFO:tasks.workunit.client.0.smithi112.stderr:166839: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:33:31.447 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:33:31.447 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:33:31.447 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:33:31.447 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:33:31.448 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:33:31.448 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:33:31.448 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:33:31.448 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:33:31.448 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:33:31.448 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:33:31.448 INFO:tasks.workunit.client.0.smithi112.stderr:166839: 1 2024-09-13T14:33:31.448 INFO:tasks.workunit.client.0.smithi112.stderr:166839: 2 2024-09-13T14:33:31.448 INFO:tasks.workunit.client.0.smithi112.stderr:166839: 3' 2024-09-13T14:33:31.448 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:33:31.448 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:31.448 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:33:31.448 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836550 2024-09-13T14:33:31.449 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836550 2024-09-13T14:33:31.449 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550' 2024-09-13T14:33:33.618 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.t.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:33.619 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:33.619 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:33:33.619 INFO:tasks.workunit.client.0.smithi112.stderr:166843: 4 2024-09-13T14:33:33.619 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:33.619 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:33:33.619 INFO:tasks.workunit.client.0.smithi112.stderr:166843: 2024-09-13T14:33:30.622+0000 7f1d2940c7c0 -1 osd.1 95 log_to_monitors true 2024-09-13T14:33:33.619 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:33.619 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:33.619 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:33.619 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:33:33.620 INFO:tasks.workunit.client.0.smithi112.stderr:166843: 5 2024-09-13T14:33:33.620 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:33.620 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:33:33.620 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:33.620 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:33.620 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:33.620 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:33:33.620 INFO:tasks.workunit.client.0.smithi112.stderr:166843: 6 2024-09-13T14:33:33.620 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:33.620 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:33:33.620 INFO:tasks.workunit.client.0.smithi112.stderr:166843: osd.1 up in weight 1 up_from 103 up_thru 103 down_at 96 last_clean_interval [93,95) [v2:127.0.0.1:6826/2442413725,v1:127.0.0.1:6827/2442413725] [v2:127.0.0.1:6828/2442413725,v1:127.0.0.1:6829/2442413725] exists,up 4b91e96b-6165-48ee-b271-78e6091f8314 2024-09-13T14:33:33.621 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:33:33.621 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:33:33.621 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:33:33.621 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:33:33.621 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:33:33.621 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:33:33.621 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:33:33.621 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:33:33.621 INFO:tasks.workunit.client.0.smithi112.stderr:166843: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:33:33.621 INFO:tasks.workunit.client.0.smithi112.stderr:166843: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:33:33.621 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:33:33.621 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:33:33.621 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:33:33.622 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:33:33.622 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:33:33.622 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:33:33.765 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/ssh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:33.765 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:33:33.765 INFO:tasks.workunit.client.0.smithi112.stderr:166839: Error ENXIO: problem getting command descriptions from osd.1 2024-09-13T14:33:33.765 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-13T14:33:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-13T14:33:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-13T14:33:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:33:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318285 2024-09-13T14:33:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318285 2024-09-13T14:33:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 2-352187318285' 2024-09-13T14:33:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:33:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=425201762306 2024-09-13T14:33:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 425201762306 2024-09-13T14:33:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 2-352187318285 3-425201762306' 2024-09-13T14:33:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:33:33.767 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836550 2024-09-13T14:33:33.767 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:33:33.767 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:33:33.767 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836550 2024-09-13T14:33:33.767 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:33:33.767 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836550 2024-09-13T14:33:33.767 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-09-13T14:33:33.767 INFO:tasks.workunit.client.0.smithi112.stderr:166839: waiting osd.0 seq 21474836550 2024-09-13T14:33:33.767 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:33:33.767 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836549 -lt 21474836550 2024-09-13T14:33:33.767 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:33:33.767 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:33:33.767 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:33:33.767 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836550 -lt 21474836550 2024-09-13T14:33:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:33:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318285 2024-09-13T14:33:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:33:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:33:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318285 2024-09-13T14:33:34.688 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtestandalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:33:34.689 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:33:34.689 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:33:34.689 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:33:34.689 INFO:tasks.workunit.client.0.smithi112.stderr:166843: 1 2024-09-13T14:33:34.689 INFO:tasks.workunit.client.0.smithi112.stderr:166843: 2 2024-09-13T14:33:34.689 INFO:tasks.workunit.client.0.smithi112.stderr:166843: 3' 2024-09-13T14:33:34.689 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:33:34.689 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:34.689 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:33:34.689 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836552 2024-09-13T14:33:34.689 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836552 2024-09-13T14:33:34.689 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552' 2024-09-13T14:33:34.689 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:34.690 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:33:34.690 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=442381631490 2024-09-13T14:33:34.690 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 442381631490 2024-09-13T14:33:34.690 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-442381631490' 2024-09-13T14:33:34.690 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:34.690 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:33:34.690 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318287 2024-09-13T14:33:34.690 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318287 2024-09-13T14:33:34.690 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-442381631490 2-352187318287' 2024-09-13T14:33:34.690 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:33:34.690 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:33:34.690 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=425201762307 2024-09-13T14:33:34.690 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 425201762307 2024-09-13T14:33:34.691 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-442381631490 2-352187318287 3-425201762307' 2024-09-13T14:33:34.691 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:33:34.691 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836552 2024-09-13T14:33:34.691 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:33:34.691 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:33:34.691 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836552 2024-09-13T14:33:34.691 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:33:34.691 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836552 2024-09-13T14:33:34.691 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836552' 2024-09-13T14:33:34.691 INFO:tasks.workunit.client.0.smithi112.stderr:166843: waiting osd.0 seq 21474836552 2024-09-13T14:33:35.511 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-st/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:33:35.512 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318285 2024-09-13T14:33:35.512 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318285' 2024-09-13T14:33:35.512 INFO:tasks.workunit.client.0.smithi112.stderr:166839: waiting osd.2 seq 352187318285 2024-09-13T14:33:35.512 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:33:35.512 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318285 -lt 352187318285 2024-09-13T14:33:35.512 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:33:35.512 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-425201762306 2024-09-13T14:33:35.512 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:33:35.512 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:33:35.512 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-425201762306 2024-09-13T14:33:35.513 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:33:35.513 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=425201762306 2024-09-13T14:33:35.513 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 425201762306' 2024-09-13T14:33:35.513 INFO:tasks.workunit.client.0.smithi112.stderr:166839: waiting osd.3 seq 425201762306 2024-09-13T14:33:35.513 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:33:35.513 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 425201762306 -lt 425201762306 2024-09-13T14:33:35.513 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:33:35.513 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:33:35.513 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:33:35.513 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:33:35.513 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:33:35.514 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:33:35.514 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:33:35.514 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:33:35.514 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:33:35.514 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:33:35.514 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:33:35.514 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:33:35.514 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:33:35.514 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:33:35.514 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:33:35.514 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 5 2024-09-13T14:33:35.514 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' -1 2024-09-13T14:33:36.859 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T14:33:36.859 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=1 2024-09-13T14:33:36.859 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:33:36.859 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:33:36.859 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:33:36.859 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:33:36.859 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:33:36.859 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:33:36.859 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:33:36.859 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:33:36.859 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:33:36.860 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:33:36.860 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T14:33:36.860 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:33:36.860 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:33:36.860 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:33:36.860 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 5 2024-09-13T14:33:36.860 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-13T14:33:36.860 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T14:33:36.860 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T14:33:36.860 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T14:33:36.860 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T14:33:36.860 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T14:33:36.861 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T14:33:36.861 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T14:33:36.861 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=3 2024-09-13T14:33:36.861 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 3 '!=' null 2024-09-13T14:33:36.861 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-13T14:33:36.861 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:33:36.861 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:33:36.861 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:33:37.589 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:33:37.589 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:33:37.589 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:33:37.589 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:33:37.589 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:33:37.589 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:33:37.589 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:33:37.589 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:33:37.589 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:33:37.589 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:33:37.590 INFO:tasks.workunit.client.0.smithi112.stderr:166839: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:33:37.590 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:33:37.590 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:33:37.590 INFO:tasks.workunit.client.0.smithi112.stderr:166839: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:33:37.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:33:37.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:33:37.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 166840 2024-09-13T14:33:38.299 INFO:tasks.workunit.client.0.smithi112.stderr:eq 0 2024-09-13T14:33:38.299 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836551 -lt 21474836552 2024-09-13T14:33:38.299 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:33:38.299 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:33:38.299 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:33:38.299 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836551 -lt 21474836552 2024-09-13T14:33:38.299 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:33:38.299 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T14:33:38.299 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:33:38.299 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836552 -lt 21474836552 2024-09-13T14:33:38.300 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:33:38.300 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-442381631490 2024-09-13T14:33:38.300 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:33:38.300 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:33:38.300 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-442381631490 2024-09-13T14:33:38.300 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:33:38.300 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=442381631490 2024-09-13T14:33:38.300 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 442381631490' 2024-09-13T14:33:38.300 INFO:tasks.workunit.client.0.smithi112.stderr:166843: waiting osd.1 seq 442381631490 2024-09-13T14:33:38.300 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:33:38.300 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 442381631490 -lt 442381631490 2024-09-13T14:33:38.300 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:33:38.300 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318287 2024-09-13T14:33:38.300 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:33:38.301 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:33:38.301 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318287 2024-09-13T14:33:38.301 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:33:38.301 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318287 2024-09-13T14:33:38.301 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318287' 2024-09-13T14:33:38.301 INFO:tasks.workunit.client.0.smithi112.stderr:166843: waiting osd.2 seq 352187318287 2024-09-13T14:33:38.301 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:33:38.301 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318287 -lt 352187318287 2024-09-13T14:33:38.301 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:33:38.301 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-425201762307 2024-09-13T14:33:38.301 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:33:38.301 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:33:39.752 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-425201762307 2024-09-13T14:33:39.752 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:33:39.752 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=425201762307 2024-09-13T14:33:39.752 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 425201762307' 2024-09-13T14:33:39.752 INFO:tasks.workunit.client.0.smithi112.stderr:166843: waiting osd.3 seq 425201762307 2024-09-13T14:33:39.752 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:33:39.752 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 425201762308 -lt 425201762307 2024-09-13T14:33:39.753 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:33:39.753 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:33:39.753 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:33:39.753 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:33:39.753 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:33:39.753 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:33:39.753 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:33:39.753 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:33:39.753 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:33:39.753 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:33:39.754 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:33:39.754 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:33:39.754 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:33:39.754 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:33:39.754 INFO:tasks.workunit.client.0.smithi112.stderr:166843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:33:39.754 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:33:39.755 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:33:39.755 INFO:tasks.workunit.client.0.smithi112.stderr:166843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:33:39.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:33:39.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:33:39.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:33:39.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:33:39.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-13T14:33:39.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:33:39.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:33:39.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:33:39.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T14:33:39.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:33:39.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:33:39.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:33:39.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:33:39.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:33:39.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:33:39.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:33:39.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:33:40.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:33:40.162 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:33:40.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:33:40.164 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:33:40.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:33:40.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:33:40.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:33:40.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:33:40.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:33:40.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:33:40.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:33:40.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:33:40.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:33:40.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:33:40.207 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:33:40.208 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:33:40.208 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:33:40.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:33:40.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:33:40.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:33:40.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T14:33:40.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T14:33:40.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T14:33:40.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T14:33:40.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:33:40.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:33:40.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:33:40.210 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:33:40.210 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:33:40.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:33:40.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:33:40.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:33:40.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:33:40.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:33:40.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:33:40.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:33:40.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:33:40.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:33:40.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:33:40.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:33:40.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:33:40.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:33:40.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:33:40.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:33:40.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:33:40.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:33:40.220 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:33:40.220 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:33:40.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:33:40.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:33:40.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:33:40.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:33:40.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T14:33:40.223 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:33:40.223 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:33:40.223 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:33:40.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T14:33:40.224 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:33:40.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T14:33:40.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:33:40.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T14:33:40.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-09-13T14:33:40.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-09-13T14:33:40.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-09-13T14:33:40.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-09-13T14:33:40.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-09-13T14:33:40.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-09-13T14:33:40.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-09-13T14:33:40.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T14:33:40.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T14:33:40.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T14:33:40.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T14:33:40.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T14:33:40.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T14:33:40.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T14:33:40.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:33:40.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:33:40.696 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:33:40.696 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:33:40.696 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:33:40.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:33:40.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T14:33:40.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T14:33:40.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T14:33:40.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:33:40.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:33:40.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T14:33:40.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T14:33:40.915 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:33:40.915 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:33:40.915 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:33:40.915 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:33:40.916 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:33:40.916 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:33:40.916 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:33:40.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:33:40.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T14:33:41.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T14:33:41.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:33:41.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:33:41.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T14:33:41.033 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T14:33:41.034 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:33:41.034 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:33:41.034 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:33:41.034 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:33:41.034 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:33:41.034 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:33:41.034 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:33:41.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:33:41.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T14:33:41.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-09-13T14:33:41.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T14:33:41.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T14:33:41.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T14:33:41.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T14:33:41.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T14:33:41.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T14:33:41.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T14:33:41.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:33:41.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:33:41.991 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:33:41.991 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:33:41.991 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:33:41.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:33:41.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:33:41.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:33:42.027 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-09-13T14:33:42.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-13T14:33:42.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-09-13T14:33:42.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:33:42.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:33:42.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T14:33:42.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:33:42.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:33:42.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:33:42.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:33:42.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:33:42.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:33:42.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:33:42.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:33:42.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:33:42.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:33:42.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:33:42.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:33:42.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:33:42.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:33:42.032 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:33:42.032 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:33:42.032 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:33:42.033 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:33:42.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:33:42.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:33:42.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:33:42.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:33:42.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:33:42.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:33:42.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:33:42.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:33:42.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:33:42.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:33:42.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:33:42.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:33:42.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:33:42.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:33:42.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ed4fad66-549c-4343-9b1e-f19eb11bf946 2024-09-13T14:33:42.034 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 ed4fad66-549c-4343-9b1e-f19eb11bf946 2024-09-13T14:33:42.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ed4fad66-549c-4343-9b1e-f19eb11bf946' 2024-09-13T14:33:42.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:33:42.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBGTeRmqXEmAxAArUzMVZxU1TEvYa2n3gfpNA== 2024-09-13T14:33:42.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBGTeRmqXEmAxAArUzMVZxU1TEvYa2n3gfpNA=="}' 2024-09-13T14:33:42.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ed4fad66-549c-4343-9b1e-f19eb11bf946 -i td/osd-scrub-repair/0/new.json 2024-09-13T14:33:42.293 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:33:42.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T14:33:42.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGTeRmqXEmAxAArUzMVZxU1TEvYa2n3gfpNA== --osd-uuid ed4fad66-549c-4343-9b1e-f19eb11bf946 2024-09-13T14:33:42.346 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:33:42.346+0000 7f3984f977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:42.351 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:33:42.350+0000 7f3984f977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:42.352 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:33:42.350+0000 7f3984f977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:42.353 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:33:42.350+0000 7f3984f977c0 -1 bdev(0x558a0ed61c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:33:42.353 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:33:42.350+0000 7f3984f977c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T14:33:48.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T14:33:48.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:33:48.744 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:33:48.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T14:33:48.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:33:49.141 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:33:49.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T14:33:49.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:33:49.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:33:49.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:33:49.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:33:49.173 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:33:49.170+0000 7feab8ea27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:49.175 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:33:49.174+0000 7feab8ea27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:49.177 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:33:49.174+0000 7feab8ea27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:33:49.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T14:33:49.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:33:49.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:33:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:33:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:33:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:33:49.481 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:33:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:49.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:33:49.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:50.664 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:33:50.662+0000 7feab8ea27c0 -1 Falling back to public interface 2024-09-13T14:33:50.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:50.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:50.847 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:33:50.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:33:50.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:50.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:33:51.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:52.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:52.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:52.183 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:33:52.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:33:52.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:52.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:33:52.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:53.430 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:33:53.430+0000 7feab8ea27c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:33:53.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:53.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:53.520 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:33:53.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:33:53.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:53.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:33:53.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:54.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:54.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:54.941 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:33:54.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:33:54.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:54.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:33:55.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:56.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:56.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:56.397 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:33:56.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:33:56.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:56.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:33:56.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:57.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:57.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:57.732 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:33:57.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:33:57.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:57.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:33:58.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:33:59.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:33:59.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:33:59.070 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:33:59.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:33:59.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:33:59.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:33:59.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:00.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:00.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:00.437 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:34:00.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:34:00.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:00.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:34:00.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:01.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:01.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:01.776 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:34:01.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:34:01.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:01.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:34:02.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:03.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:03.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:03.113 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:34:03.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:34:03.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:03.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:34:03.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:04.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:04.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:04.454 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:34:04.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:34:04.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:04.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:34:04.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:05.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:05.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:05.805 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:34:05.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:34:05.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:05.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:34:06.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:07.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:07.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:07.222 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:34:07.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:34:07.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:07.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:34:07.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:08.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:08.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:08.837 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:34:08.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:34:08.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:08.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:34:09.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:10.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:10.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:10.793 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:34:10.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:34:10.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:10.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:34:11.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:12.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:12.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:12.646 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:34:12.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:34:12.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:12.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:34:13.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:14.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:14.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:14.137 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:34:14.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:34:14.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:14.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:34:14.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:15.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:15.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:15.944 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:34:15.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:34:15.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:15.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:34:16.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:17.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:17.638 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:34:17.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:17.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:34:17.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:17.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:34:17.974 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1605190593,v1:127.0.0.1:6803/1605190593] [v2:127.0.0.1:6804/1605190593,v1:127.0.0.1:6805/1605190593] exists,up ed4fad66-549c-4343-9b1e-f19eb11bf946 2024-09-13T14:34:17.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:34:17.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:34:17.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:34:17.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-13T14:34:17.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-09-13T14:34:17.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:34:17.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:34:17.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T14:34:17.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:34:17.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:34:17.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:34:17.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:34:17.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:34:17.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:34:17.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:34:17.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:34:17.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:34:17.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:34:17.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:34:17.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:34:17.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:34:17.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:34:17.979 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:34:17.980 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:34:17.980 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:34:17.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:34:17.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:34:17.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:34:17.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:34:17.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:34:17.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:34:17.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:34:17.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:34:17.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:34:17.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:34:17.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:34:17.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:34:17.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:34:17.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:34:17.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:34:17.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6bbc9a9f-3432-44d8-a504-eec0cac352ac 2024-09-13T14:34:17.981 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 6bbc9a9f-3432-44d8-a504-eec0cac352ac 2024-09-13T14:34:17.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 6bbc9a9f-3432-44d8-a504-eec0cac352ac' 2024-09-13T14:34:17.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:34:18.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBqTeRmq+ctABAAYb8aHUig9tm/DMlNh6/zmQ== 2024-09-13T14:34:18.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBqTeRmq+ctABAAYb8aHUig9tm/DMlNh6/zmQ=="}' 2024-09-13T14:34:18.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6bbc9a9f-3432-44d8-a504-eec0cac352ac -i td/osd-scrub-repair/1/new.json 2024-09-13T14:34:18.455 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:34:18.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T14:34:18.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBqTeRmq+ctABAAYb8aHUig9tm/DMlNh6/zmQ== --osd-uuid 6bbc9a9f-3432-44d8-a504-eec0cac352ac 2024-09-13T14:34:18.509 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:34:18.505+0000 7f447994f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:34:18.539 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:34:18.537+0000 7f447994f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:34:18.541 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:34:18.537+0000 7f447994f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:34:18.541 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:34:18.537+0000 7f447994f7c0 -1 bdev(0x559285a5fc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:34:18.541 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:34:18.537+0000 7f447994f7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T14:34:24.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T14:34:24.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:34:24.669 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:34:24.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T14:34:24.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:34:25.068 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:34:25.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T14:34:25.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:34:25.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:34:25.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:34:25.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:34:25.100 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:34:25.097+0000 7fde05e377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:34:25.119 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:34:25.117+0000 7fde05e377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:34:25.120 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:34:25.117+0000 7fde05e377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:34:25.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T14:34:25.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:34:25.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:34:25.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:34:25.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:34:25.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:25.398 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:34:25.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:34:25.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:25.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:25.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:26.184 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:34:26.181+0000 7fde05e377c0 -1 Falling back to public interface 2024-09-13T14:34:26.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:26.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:26.752 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:34:26.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:34:26.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:26.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:27.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:28.080 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:34:28.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:34:28.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:28.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:28.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:29.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:29.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:29.405 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:34:29.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:34:29.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:29.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:29.512 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:34:29.509+0000 7fde05e377c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:34:29.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:30.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:30.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:30.736 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:34:30.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:34:30.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:30.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:31.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:32.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:32.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:32.076 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:34:32.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:34:32.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:32.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:32.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:33.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:33.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:33.567 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:34:33.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:34:33.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:33.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:33.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:34.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:34.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:34.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:34:34.899 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:34:34.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:34.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:35.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:36.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:36.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:36.373 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:34:36.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:34:36.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:36.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:36.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:37.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:37.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:37.715 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:34:37.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:34:37.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:37.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:39.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:39.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:39.058 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:34:39.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:34:39.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:39.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:39.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:40.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:40.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:40.554 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:34:40.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:34:40.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:40.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:40.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:41.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:41.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:41.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:34:41.883 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:34:41.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:41.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:42.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:43.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:43.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:43.206 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:34:43.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:34:43.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:43.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:44.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:45.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:45.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:45.060 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:34:45.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:34:45.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:45.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:45.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:46.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:46.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:46.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:34:46.865 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:34:46.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:46.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:47.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:48.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:48.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:48.634 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:34:48.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:34:48.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:48.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:49.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:50.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:50.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:50.750 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:34:50.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:34:50.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:50.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:51.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:52.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:52.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:52.586 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:34:52.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:34:52.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:52.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:52.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:34:53.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:34:53.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:34:53.911 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:34:53.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:34:53.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:34:53.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:34:54.246 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2523904545,v1:127.0.0.1:6811/2523904545] [v2:127.0.0.1:6812/2523904545,v1:127.0.0.1:6813/2523904545] exists,up 6bbc9a9f-3432-44d8-a504-eec0cac352ac 2024-09-13T14:34:54.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:34:54.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:34:54.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:34:54.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-13T14:34:54.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-09-13T14:34:54.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:34:54.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:34:54.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T14:34:54.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:34:54.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:34:54.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:34:54.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:34:54.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:34:54.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:34:54.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T14:34:54.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T14:34:54.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:34:54.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:34:54.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:34:54.250 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:34:54.250 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:34:54.250 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:34:54.250 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:34:54.251 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:34:54.251 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:34:54.251 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:34:54.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:34:54.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:34:54.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:34:54.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:34:54.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:34:54.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:34:54.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:34:54.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:34:54.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:34:54.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:34:54.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:34:54.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:34:54.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T14:34:54.252 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:34:54.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c38afcf9-9793-45de-9f1c-b2f5c8d69cc9 2024-09-13T14:34:54.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c38afcf9-9793-45de-9f1c-b2f5c8d69cc9' 2024-09-13T14:34:54.253 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 c38afcf9-9793-45de-9f1c-b2f5c8d69cc9 2024-09-13T14:34:54.253 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:34:54.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCOTeRm1NdZEBAAZutSlt6hCnrQibkK4MjvhQ== 2024-09-13T14:34:54.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCOTeRm1NdZEBAAZutSlt6hCnrQibkK4MjvhQ=="}' 2024-09-13T14:34:54.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c38afcf9-9793-45de-9f1c-b2f5c8d69cc9 -i td/osd-scrub-repair/2/new.json 2024-09-13T14:34:54.685 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:34:54.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T14:34:54.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCOTeRm1NdZEBAAZutSlt6hCnrQibkK4MjvhQ== --osd-uuid c38afcf9-9793-45de-9f1c-b2f5c8d69cc9 2024-09-13T14:34:54.739 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:34:54.736+0000 7f19490107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:34:54.802 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:34:54.804+0000 7f19490107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:34:54.803 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:34:54.804+0000 7f19490107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:34:54.804 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:34:54.804+0000 7f19490107c0 -1 bdev(0x5610140ffc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T14:34:54.804 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:34:54.804+0000 7f19490107c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T14:35:01.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T14:35:01.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:35:01.188 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T14:35:01.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T14:35:01.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:35:01.600 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T14:35:01.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T14:35:01.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:35:01.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:35:01.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:35:01.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:35:01.633 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:35:01.632+0000 7f2e5a6937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:35:01.659 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:35:01.660+0000 7f2e5a6937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:35:01.660 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:35:01.660+0000 7f2e5a6937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:35:01.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T14:35:01.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:35:01.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:35:01.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:35:01.923 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:35:01.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:35:01.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:01.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:35:01.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:01.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:02.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:02.960 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:35:02.960+0000 7f2e5a6937c0 -1 Falling back to public interface 2024-09-13T14:35:03.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:03.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:03.270 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:35:03.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:35:03.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:03.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:03.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:04.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:04.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:04.607 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:35:04.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:35:04.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:04.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:04.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:05.758 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:35:05.760+0000 7f2e5a6937c0 -1 osd.2 0 log_to_monitors true 2024-09-13T14:35:05.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:05.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:05.952 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:35:05.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:35:05.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:05.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:06.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:07.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:07.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:07.299 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:35:07.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:35:07.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:07.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:07.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:08.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:08.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:08.792 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:35:08.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:35:08.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:08.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:09.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:10.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:10.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:10.208 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:35:10.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:35:10.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:10.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:10.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:11.622 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:35:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:35:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:11.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:12.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:12.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:12.960 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:35:12.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:35:12.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:12.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:13.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:14.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:14.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:14.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:35:14.406 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:35:14.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:14.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:14.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:15.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:15.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:15.800 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:35:15.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:35:15.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:15.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:16.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:17.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:17.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:17.142 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:35:17.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:35:17.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:17.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:17.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:18.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:18.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:18.734 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:35:18.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:35:18.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:18.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:19.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:20.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:20.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:20.366 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:35:20.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:35:20.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:20.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:20.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:21.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:21.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:21.843 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:35:21.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:35:21.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:21.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:22.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:23.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:23.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:23.402 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:35:23.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:35:23.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:23.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:24.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:25.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:25.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:25.210 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:35:25.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:35:25.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:25.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:26.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:27.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:27.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:27.218 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:35:27.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:35:27.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:27.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:27.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:28.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:28.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:28.822 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:35:28.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:35:28.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:28.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:29.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:30.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:30.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:30.713 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:35:30.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:35:30.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:30.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:35:31.043 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2875573193,v1:127.0.0.1:6819/2875573193] [v2:127.0.0.1:6820/2875573193,v1:127.0.0.1:6821/2875573193] exists,up c38afcf9-9793-45de-9f1c-b2f5c8d69cc9 2024-09-13T14:35:31.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:35:31.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:35:31.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:35:31.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-13T14:35:31.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-09-13T14:35:31.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:35:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:35:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-13T14:35:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:35:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:35:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:35:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:35:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:35:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:35:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:35:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:35:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:35:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:35:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:35:31.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:35:31.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:35:31.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:35:31.048 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:35:31.048 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:35:31.048 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:35:31.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:35:31.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:35:31.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:35:31.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:35:31.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:35:31.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:35:31.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:35:31.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:35:31.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:35:31.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:35:31.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:35:31.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:35:31.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:35:31.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:35:31.050 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:35:31.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0c26729b-baf1-4c78-9d61-9982d2d70dc2 2024-09-13T14:35:31.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 0c26729b-baf1-4c78-9d61-9982d2d70dc2' 2024-09-13T14:35:31.050 INFO:tasks.workunit.client.0.smithi112.stdout:add osd3 0c26729b-baf1-4c78-9d61-9982d2d70dc2 2024-09-13T14:35:31.050 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:35:31.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCzTeRmFrhfBBAAb/Ksx8+NErm451Sj2jeKPw== 2024-09-13T14:35:31.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCzTeRmFrhfBBAAb/Ksx8+NErm451Sj2jeKPw=="}' 2024-09-13T14:35:31.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0c26729b-baf1-4c78-9d61-9982d2d70dc2 -i td/osd-scrub-repair/3/new.json 2024-09-13T14:35:31.441 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:35:31.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-13T14:35:31.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCzTeRmFrhfBBAAb/Ksx8+NErm451Sj2jeKPw== --osd-uuid 0c26729b-baf1-4c78-9d61-9982d2d70dc2 2024-09-13T14:35:31.496 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:35:31.495+0000 7f29b506f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:35:31.526 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:35:31.527+0000 7f29b506f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:35:31.527 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:35:31.527+0000 7f29b506f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:35:31.528 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:35:31.527+0000 7f29b506f7c0 -1 bdev(0x560e37bffc00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-13T14:35:31.528 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:35:31.527+0000 7f29b506f7c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-13T14:35:38.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-13T14:35:38.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:35:38.379 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd3 key to auth repository 2024-09-13T14:35:38.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-13T14:35:38.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:35:38.777 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.3 2024-09-13T14:35:38.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-13T14:35:38.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:35:38.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:35:38.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:35:38.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:35:38.808 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:35:38.807+0000 7f0eb9dbc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:35:38.810 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:35:38.811+0000 7f0eb9dbc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:35:38.811 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:35:38.811+0000 7f0eb9dbc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:35:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-13T14:35:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:35:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:35:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:35:39.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:35:39.110 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:35:39.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:39.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:35:39.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:39.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:35:39.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:40.117 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:35:40.115+0000 7f0eb9dbc7c0 -1 Falling back to public interface 2024-09-13T14:35:40.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:40.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:40.420 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:35:40.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:35:40.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:40.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:35:40.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:41.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:41.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:41.731 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:35:41.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:35:41.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:41.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:35:42.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:43.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:43.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:43.047 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:35:43.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:35:43.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:43.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:35:43.228 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:35:43.227+0000 7f0eb9dbc7c0 -1 osd.3 0 log_to_monitors true 2024-09-13T14:35:43.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:44.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:44.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:44.368 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:35:44.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:35:44.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:44.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:35:44.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:45.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:45.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:45.706 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:35:45.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:35:45.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:45.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:35:46.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:47.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:47.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:47.180 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:35:47.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:35:47.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:47.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:35:47.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:48.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:48.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:48.499 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:35:48.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:35:48.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:48.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:35:48.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:49.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:49.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:49.832 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:35:49.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:35:49.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:49.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:35:50.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:51.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:51.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:51.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:35:51.217 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:35:51.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:51.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:35:51.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:52.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:52.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:52.589 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:35:52.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:35:52.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:52.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:35:52.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:53.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:53.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:53.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:35:53.992 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:35:53.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:53.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:35:54.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:55.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:55.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:55.412 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:35:55.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:35:55.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:55.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:35:55.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:56.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:56.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:56.819 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:35:56.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:35:56.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:56.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:35:57.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:58.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:58.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:58.234 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:35:58.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:35:58.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:58.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:35:58.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:35:59.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:35:59.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:35:59.732 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:35:59.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:35:59.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:35:59.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:36:00.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:36:01.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:36:01.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:36:01.485 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:36:01.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:36:01.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:36:01.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:36:02.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:36:03.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:36:03.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:36:03.285 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:36:03.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:36:03.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:36:03.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:36:04.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:36:05.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:36:05.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:36:05.039 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:36:05.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:36:05.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:36:05.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:36:05.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:36:06.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:36:06.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:36:06.990 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:36:06.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:36:06.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:36:06.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:36:07.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:36:08.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:36:08.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:36:08.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T14:36:08.578 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T14:36:08.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:36:08.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:36:08.913 INFO:tasks.workunit.client.0.smithi112.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/311315272,v1:127.0.0.1:6827/311315272] [v2:127.0.0.1:6828/311315272,v1:127.0.0.1:6829/311315272] exists,up 0c26729b-baf1-4c78-9d61-9982d2d70dc2 2024-09-13T14:36:08.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:36:08.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:36:08.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:36:08.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-09-13T14:36:08.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T14:36:09.280 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' does not exist 2024-09-13T14:36:09.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T14:36:09.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T14:36:09.728 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T14:36:09.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:36:10.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T14:36:13.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-09-13T14:36:13.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:36:13.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:36:13.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:36:13.241 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:36:13.242 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:36:13.242 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:36:13.243 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:36:13.243 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:36:13.243 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:36:13.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:36:13.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:36:13.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:36:13.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:36:13.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:36:13.371 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:36:13.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:36:13.780 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:36:13.781 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:36:13.781 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T14:36:13.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:36:13.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:13.781 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:36:13.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836512 2024-09-13T14:36:13.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836512 2024-09-13T14:36:13.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512' 2024-09-13T14:36:13.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:13.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:36:14.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672981 2024-09-13T14:36:14.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672981 2024-09-13T14:36:14.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672981' 2024-09-13T14:36:14.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:14.078 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:36:14.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509453 2024-09-13T14:36:14.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509453 2024-09-13T14:36:14.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672981 2-64424509453' 2024-09-13T14:36:14.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:14.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:36:14.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345928 2024-09-13T14:36:14.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345928 2024-09-13T14:36:14.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672981 2-64424509453 3-85899345928' 2024-09-13T14:36:14.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:36:14.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836512 2024-09-13T14:36:14.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:36:14.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:36:14.361 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836512 2024-09-13T14:36:14.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:36:14.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836512 2024-09-13T14:36:14.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-13T14:36:14.363 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836512 2024-09-13T14:36:14.363 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:36:14.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836512 2024-09-13T14:36:14.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:36:15.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:36:15.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:36:15.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-13T14:36:15.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:36:15.992 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672981 2024-09-13T14:36:15.992 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:36:15.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:36:15.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672981 2024-09-13T14:36:15.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:36:15.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672981 2024-09-13T14:36:15.996 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672981 2024-09-13T14:36:15.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-13T14:36:15.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:36:16.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-13T14:36:16.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:36:16.320 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509453 2024-09-13T14:36:16.320 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:36:16.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:36:16.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509453 2024-09-13T14:36:16.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:36:16.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509453 2024-09-13T14:36:16.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-13T14:36:16.324 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509453 2024-09-13T14:36:16.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:36:16.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-13T14:36:16.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:36:16.666 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345928 2024-09-13T14:36:16.666 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:36:16.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:36:16.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345928 2024-09-13T14:36:16.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:36:16.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345928 2024-09-13T14:36:16.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2024-09-13T14:36:16.670 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.3 seq 85899345928 2024-09-13T14:36:16.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:36:17.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345928 -lt 85899345928 2024-09-13T14:36:17.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:36:17.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:36:17.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:36:17.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T14:36:17.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:36:17.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:36:17.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:36:17.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:36:17.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:36:17.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:36:17.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:36:17.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:36:17.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:36:17.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:36:17.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:36:18.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T14:36:18.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:36:18.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:36:18.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-09-13T14:36:18.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-13T14:36:18.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-13T14:36:18.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-13T14:36:18.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-13T14:36:18.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-13T14:36:18.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-13T14:36:18.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-13T14:36:19.391 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'ecpool' created 2024-09-13T14:36:19.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:36:20.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-13T14:36:20.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-13T14:36:21.025 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 2 allow_ec_overwrites to true 2024-09-13T14:36:21.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-13T14:36:21.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:36:21.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:36:21.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:36:21.046 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:36:21.047 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:36:21.047 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:36:21.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:36:21.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:36:21.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:36:21.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:36:21.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:36:21.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:36:21.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:36:21.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:36:21.161 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:36:21.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:36:21.499 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:36:21.499 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:36:21.500 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T14:36:21.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:36:21.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:21.500 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:36:21.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836514 2024-09-13T14:36:21.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836514 2024-09-13T14:36:21.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514' 2024-09-13T14:36:21.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:21.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:36:21.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672983 2024-09-13T14:36:21.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672983 2024-09-13T14:36:21.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-42949672983' 2024-09-13T14:36:21.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:21.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:36:21.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509456 2024-09-13T14:36:21.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509456 2024-09-13T14:36:21.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-42949672983 2-64424509456' 2024-09-13T14:36:21.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:21.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:36:22.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345930 2024-09-13T14:36:22.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345930 2024-09-13T14:36:22.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-42949672983 2-64424509456 3-85899345930' 2024-09-13T14:36:22.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:36:22.121 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836514 2024-09-13T14:36:22.121 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:36:22.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:36:22.122 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836514 2024-09-13T14:36:22.122 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:36:22.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836514 2024-09-13T14:36:22.123 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836514 2024-09-13T14:36:22.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-09-13T14:36:22.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:36:22.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836513 -lt 21474836514 2024-09-13T14:36:22.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:36:23.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:36:23.430 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:36:23.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836514 -lt 21474836514 2024-09-13T14:36:23.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:36:23.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672983 2024-09-13T14:36:23.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:36:23.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:36:23.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672983 2024-09-13T14:36:23.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:36:23.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672983 2024-09-13T14:36:23.765 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672983 2024-09-13T14:36:23.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-13T14:36:23.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:36:24.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-13T14:36:24.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:36:24.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509456 2024-09-13T14:36:24.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:36:24.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:36:24.096 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509456 2024-09-13T14:36:24.096 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:36:24.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509456 2024-09-13T14:36:24.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-13T14:36:24.098 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509456 2024-09-13T14:36:24.098 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:36:24.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-13T14:36:24.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:36:24.429 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345930 2024-09-13T14:36:24.429 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:36:24.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:36:24.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345930 2024-09-13T14:36:24.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:36:24.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345930 2024-09-13T14:36:24.433 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.3 seq 85899345930 2024-09-13T14:36:24.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2024-09-13T14:36:24.433 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:36:24.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345930 -lt 85899345930 2024-09-13T14:36:24.786 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:36:24.786 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:36:24.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:36:25.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:36:25.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:36:25.230 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:36:25.230 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:36:25.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:36:25.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:36:25.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:36:25.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:36:25.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:36:25.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:36:25.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:36:25.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:36:26.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:36:26.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:36:26.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:36:26.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-13T14:36:26.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-13T14:36:26.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-13T14:36:26.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-13T14:36:26.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-13T14:36:26.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T14:36:26.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T14:36:26.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-13T14:36:26.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T14:36:26.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T14:36:26.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T14:36:26.494 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T14:36:26.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T14:36:26.980 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T14:36:26.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T14:36:26.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T14:36:27.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-13T14:36:27.439 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-13T14:36:27.439 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-13T14:36:27.439 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:36:27.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:36:27.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:36:27.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-13T14:36:27.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-13T14:36:27.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-13T14:36:27.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-13T14:36:27.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-13T14:36:27.764 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:36:27.765 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T14:36:28.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-13T14:36:28.093 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:36:28.093 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:36:28.093 INFO:tasks.workunit.client.0.smithi112.stderr:0' 2024-09-13T14:36:28.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 0 2024-09-13T14:36:28.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-09-13T14:36:28.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-13T14:36:28.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-09-13T14:36:28.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-09-13T14:36:28.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-13T14:36:28.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-13T14:36:28.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-13T14:36:28.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-13T14:36:28.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:36:28.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:36:28.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:36:28.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:36:28.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:36:28.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:36:28.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:36:28.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:36:28.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:36:28.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:36:28.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:36:28.096 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:36:28.096 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:36:28.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:36:28.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:36:28.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:36:28.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:36:28.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:36:28.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:36:28.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:36:28.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:36:28.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:36:28.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:36:28.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-13T14:36:30.650 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-13T14:36:31.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:36:31.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:36:31.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:36:31.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:36:31.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:36:31.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:36:31.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:36:31.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:36:31.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:36:31.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:36:31.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:36:31.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:36:31.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:36:31.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:36:31.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:36:31.195 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:36:31.195 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:36:31.195 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:36:31.196 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:36:31.196 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:36:31.196 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:36:31.196 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:36:31.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:36:31.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:36:31.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:36:31.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:36:31.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:36:31.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:36:31.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:36:31.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:36:31.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:36:31.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:36:31.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:36:31.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:36:31.198 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.3 2024-09-13T14:36:31.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:36:31.198 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:36:31.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:36:31.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:36:31.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:36:31.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:36:31.232 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:36:31.229+0000 7f4dc6c0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:36:31.235 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:36:31.233+0000 7f4dc6c0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:36:31.236 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:36:31.233+0000 7f4dc6c0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:36:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:36:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:36:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:36:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:36:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:36:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:36:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:36:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:36:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:36:31.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:36:31.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:36:32.281 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:36:32.277+0000 7f4dc6c0e7c0 -1 Falling back to public interface 2024-09-13T14:36:32.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:36:32.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:36:32.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:36:32.890 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:36:32.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:36:32.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:36:33.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:36:34.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:36:34.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:36:34.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:36:34.225 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:36:34.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:36:34.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:36:34.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:36:35.245 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:36:35.241+0000 7f4dc6c0e7c0 -1 osd.3 38 log_to_monitors true 2024-09-13T14:36:35.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:36:35.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:36:35.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:36:35.565 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:36:35.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:36:35.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:36:35.975 INFO:tasks.workunit.client.0.smithi112.stderr:osd.3 up in weight 1 up_from 43 up_thru 33 down_at 39 last_clean_interval [20,38) [v2:127.0.0.1:6826/2880650055,v1:127.0.0.1:6827/2880650055] [v2:127.0.0.1:6828/2880650055,v1:127.0.0.1:6829/2880650055] exists,up 0c26729b-baf1-4c78-9d61-9982d2d70dc2 2024-09-13T14:36:35.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:36:35.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:36:35.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:36:35.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:36:35.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:36:35.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:36:35.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:36:35.976 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:36:35.976 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:36:35.976 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:36:35.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:36:35.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:36:35.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:36:36.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:36:36.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:36:36.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:36:36.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:36:36.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:36:36.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:36:36.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:36:36.723 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:36:36.723 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:36:36.723 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T14:36:36.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:36:36.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:36.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:36:37.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836518 2024-09-13T14:36:37.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836518 2024-09-13T14:36:37.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836518' 2024-09-13T14:36:37.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:37.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:36:37.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672987 2024-09-13T14:36:37.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672987 2024-09-13T14:36:37.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836518 1-42949672987' 2024-09-13T14:36:37.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:37.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:36:37.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509460 2024-09-13T14:36:37.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509460 2024-09-13T14:36:37.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836518 1-42949672987 2-64424509460' 2024-09-13T14:36:37.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:37.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:36:37.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=184683593730 2024-09-13T14:36:37.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 184683593730 2024-09-13T14:36:37.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836518 1-42949672987 2-64424509460 3-184683593730' 2024-09-13T14:36:37.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:36:37.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836518 2024-09-13T14:36:37.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:36:37.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:36:37.591 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836518 2024-09-13T14:36:37.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:36:37.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836518 2024-09-13T14:36:37.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-09-13T14:36:37.593 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836518 2024-09-13T14:36:37.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:36:38.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836517 -lt 21474836518 2024-09-13T14:36:38.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:36:39.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:36:39.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:36:39.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836519 -lt 21474836518 2024-09-13T14:36:39.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:36:39.359 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672987 2024-09-13T14:36:39.359 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:36:39.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:36:39.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672987 2024-09-13T14:36:39.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:36:39.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672987 2024-09-13T14:36:39.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-13T14:36:39.363 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672987 2024-09-13T14:36:39.364 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:36:39.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672987 -lt 42949672987 2024-09-13T14:36:39.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:36:39.694 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509460 2024-09-13T14:36:39.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:36:39.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:36:39.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:36:39.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509460 2024-09-13T14:36:39.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509460 2024-09-13T14:36:39.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-13T14:36:39.699 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509460 2024-09-13T14:36:39.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:36:40.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509460 -lt 64424509460 2024-09-13T14:36:40.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:36:40.060 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-184683593730 2024-09-13T14:36:40.061 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:36:40.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:36:40.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-184683593730 2024-09-13T14:36:40.064 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:36:40.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=184683593730 2024-09-13T14:36:40.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2024-09-13T14:36:40.065 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 184683593730 2024-09-13T14:36:40.065 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:36:40.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 184683593730 -lt 184683593730 2024-09-13T14:36:40.388 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:36:40.388 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:36:40.388 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:36:40.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:36:40.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:36:40.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:36:40.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:36:40.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:36:40.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:36:40.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:36:40.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:36:41.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:36:41.137 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:36:41.138 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:36:41.138 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:36:41.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:36:41.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:36:41.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:36:41.509 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-13T14:36:41.509 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:36:41.509 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:36:41.509 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:36:41.509 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:36:41.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-13T14:36:41.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-13T14:36:41.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:36:41.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:36:41.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:36:41.850 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:36:41.850 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:36:41.850 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:36:42.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:36:19.242671+0000 2024-09-13T14:36:42.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:36:42.487 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:36:42.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:36:19.242671+0000 2024-09-13T14:36:42.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:36:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:36:19.242671+0000 2024-09-13T14:36:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:36:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:36:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:36:42.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:36:42.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:36:42.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:36:42.506 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:36:42.506 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:36:42.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:36:19.242671+0000 '>' 2024-09-13T14:36:19.242671+0000 2024-09-13T14:36:42.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:36:43.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:36:43.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:36:43.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:36:43.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:36:43.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:36:43.834 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:36:43.834 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:36:44.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:36:19.242671+0000 '>' 2024-09-13T14:36:19.242671+0000 2024-09-13T14:36:44.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:36:45.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:36:45.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:36:45.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:36:45.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:36:45.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:36:45.159 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:36:45.159 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:36:45.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:36:19.242671+0000 '>' 2024-09-13T14:36:19.242671+0000 2024-09-13T14:36:45.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:36:46.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:36:46.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:36:46.475 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:36:46.475 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:36:46.475 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:36:46.476 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:36:46.476 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:36:46.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:36:43.237087+0000 '>' 2024-09-13T14:36:19.242671+0000 2024-09-13T14:36:46.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:36:46.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:36:48.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:36:48.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:36:48.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:36:48.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:36:48.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:36:48.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:36:48.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:36:48.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:36:48.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:36:48.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:36:48.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:36:48.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:36:48.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:36:48.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:36:48.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:36:48.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:36:50.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:36:50.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:36:50.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:36:50.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:36:50.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:36:50.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:36:50.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:36:50.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-13T14:36:50.621 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-13T14:36:50.622 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:36:50.622 INFO:tasks.workunit.client.0.smithi112.stdout:hinfo_key 2024-09-13T14:36:50.622 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:36:50.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:36:50.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:36:50.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:36:50.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:36:50.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:36:50.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:36:50.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:36:50.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:36:50.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:36:50.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:36:50.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:36:50.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:36:50.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:36:50.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:36:50.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:36:50.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:36:50.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:36:50.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:36:50.923 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:36:50.923 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:36:50.923 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:36:50.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:36:50.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:36:50.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:36:50.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:36:50.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:36:50.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:36:50.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:36:50.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:36:50.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:36:50.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:36:50.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:36:50.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:36:50.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:36:50.924 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.3 2024-09-13T14:36:50.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:36:50.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:36:50.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:36:50.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:36:50.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:36:50.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:36:50.958 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:36:50.952+0000 7fa0626de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:36:50.960 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:36:50.956+0000 7fa0626de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:36:50.962 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:36:50.956+0000 7fa0626de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:36:51.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:36:51.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:36:51.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:36:51.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:36:51.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:36:51.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:36:51.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:36:51.264 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:36:51.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:36:51.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:36:51.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:36:52.208 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:36:52.200+0000 7fa0626de7c0 -1 Falling back to public interface 2024-09-13T14:36:52.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:36:52.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:36:52.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:36:52.600 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:36:52.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:36:52.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:36:52.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:36:53.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:36:53.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:36:53.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:36:53.928 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:36:53.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:36:53.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:36:54.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:36:55.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:36:55.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:36:55.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:36:55.254 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:36:55.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:36:55.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:36:55.422 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:36:55.420+0000 7fa0626de7c0 -1 osd.3 45 log_to_monitors true 2024-09-13T14:36:55.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:36:56.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:36:56.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:36:56.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:36:56.573 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:36:56.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:36:56.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:36:56.922 INFO:tasks.workunit.client.0.smithi112.stderr:osd.3 up in weight 1 up_from 50 up_thru 50 down_at 46 last_clean_interval [43,45) [v2:127.0.0.1:6826/1262859210,v1:127.0.0.1:6827/1262859210] [v2:127.0.0.1:6828/1262859210,v1:127.0.0.1:6829/1262859210] exists,up 0c26729b-baf1-4c78-9d61-9982d2d70dc2 2024-09-13T14:36:56.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:36:56.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:36:56.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:36:56.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:36:56.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:36:56.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:36:56.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:36:56.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:36:56.923 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:36:56.923 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:36:56.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:36:56.924 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:36:56.924 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:36:57.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:36:57.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:36:57.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:36:57.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:36:57.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:36:57.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:36:57.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:36:57.421 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:36:57.421 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:36:57.421 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T14:36:57.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:36:57.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:57.421 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:36:57.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836523 2024-09-13T14:36:57.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836523 2024-09-13T14:36:57.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836523' 2024-09-13T14:36:57.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:57.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:36:57.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672992 2024-09-13T14:36:57.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672992 2024-09-13T14:36:57.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836523 1-42949672992' 2024-09-13T14:36:57.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:57.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:36:57.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509465 2024-09-13T14:36:57.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509465 2024-09-13T14:36:57.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836523 1-42949672992 2-64424509465' 2024-09-13T14:36:57.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:36:57.850 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:36:57.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364802 2024-09-13T14:36:57.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364802 2024-09-13T14:36:57.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836523 1-42949672992 2-64424509465 3-214748364802' 2024-09-13T14:36:57.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:36:57.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836523 2024-09-13T14:36:57.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:36:57.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:36:57.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836523 2024-09-13T14:36:57.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:36:57.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836523 2024-09-13T14:36:57.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-09-13T14:36:57.987 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836523 2024-09-13T14:36:57.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:36:58.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836522 -lt 21474836523 2024-09-13T14:36:58.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:36:59.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:36:59.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:36:59.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836523 2024-09-13T14:36:59.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:36:59.621 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672992 2024-09-13T14:36:59.621 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:36:59.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:36:59.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672992 2024-09-13T14:36:59.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:36:59.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672992 2024-09-13T14:36:59.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-09-13T14:36:59.625 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672992 2024-09-13T14:36:59.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:36:59.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672992 -lt 42949672992 2024-09-13T14:36:59.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:36:59.998 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509465 2024-09-13T14:36:59.998 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:36:59.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:37:00.000 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509465 2024-09-13T14:37:00.000 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:00.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509465 2024-09-13T14:37:00.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-09-13T14:37:00.002 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509465 2024-09-13T14:37:00.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:37:00.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509465 -lt 64424509465 2024-09-13T14:37:00.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:00.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-214748364802 2024-09-13T14:37:00.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:00.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:37:00.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-214748364802 2024-09-13T14:37:00.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:00.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364802 2024-09-13T14:37:00.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 214748364802' 2024-09-13T14:37:00.381 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 214748364802 2024-09-13T14:37:00.381 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:37:00.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364802 -lt 214748364802 2024-09-13T14:37:00.704 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:37:00.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:37:00.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:37:01.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:37:01.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:37:01.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:37:01.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:37:01.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:37:01.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:37:01.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:37:01.167 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:37:01.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:37:01.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:37:01.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:37:01.497 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:37:01.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:37:01.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:37:01.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:37:01.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:37:01.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:37:01.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-09-13T14:37:01.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-13T14:37:01.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-13T14:37:01.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-09-13T14:37:01.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:37:01.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:37:01.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:37:01.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:37:01.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:37:01.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:37:01.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:37:01.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:37:01.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:37:01.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:37:01.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:37:01.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:37:01.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:37:01.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:37:01.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:37:01.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:37:02.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:37:02.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:37:02.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:37:02.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:37:02.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:37:02.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:37:02.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:37:02.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-13T14:37:04.281 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-13T14:37:04.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T14:37:04.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:37:04.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:37:04.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:37:04.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:37:04.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:37:04.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:37:04.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:37:04.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:37:04.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:37:04.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:37:04.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:37:04.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:37:04.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:37:04.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:37:04.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:37:04.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:37:04.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:37:04.819 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:37:04.819 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:37:04.819 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:37:04.819 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:37:04.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:37:04.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:37:04.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:37:04.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:37:04.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:37:04.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:37:04.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:37:04.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:37:04.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:37:04.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:37:04.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:37:04.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:37:04.821 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T14:37:04.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:37:04.821 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:37:04.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:37:04.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:37:04.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:37:04.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:37:04.855 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:37:04.856+0000 7f346f3127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:37:04.858 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:37:04.860+0000 7f346f3127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:37:04.860 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:37:04.860+0000 7f346f3127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:37:05.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:37:05.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:37:05.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:37:05.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:37:05.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:37:05.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:05.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:37:05.174 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:37:05.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:05.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:37:05.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:06.157 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:37:06.156+0000 7f346f3127c0 -1 Falling back to public interface 2024-09-13T14:37:06.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:06.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:06.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:37:06.490 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:37:06.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:06.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:37:06.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:07.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:07.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:07.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:37:07.813 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:37:07.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:07.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:37:08.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:08.943 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:37:08.944+0000 7f346f3127c0 -1 osd.1 52 log_to_monitors true 2024-09-13T14:37:09.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:09.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:09.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:37:09.137 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:37:09.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:09.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:37:09.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:10.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:10.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:10.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:37:10.491 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:37:10.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:10.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:37:10.832 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [10,52) [v2:127.0.0.1:6810/2569389153,v1:127.0.0.1:6811/2569389153] [v2:127.0.0.1:6812/2569389153,v1:127.0.0.1:6813/2569389153] exists,up 6bbc9a9f-3432-44d8-a504-eec0cac352ac 2024-09-13T14:37:10.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:37:10.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:37:10.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:37:10.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:37:10.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:37:10.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:37:10.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:37:10.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:37:10.834 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:37:10.834 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:37:10.835 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:37:10.835 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:37:10.835 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:37:10.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:37:10.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:37:10.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:37:10.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:37:10.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:37:10.941 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:37:11.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:37:11.378 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:37:11.378 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:37:11.378 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T14:37:11.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:37:11.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:11.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:37:11.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836527 2024-09-13T14:37:11.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836527 2024-09-13T14:37:11.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527' 2024-09-13T14:37:11.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:11.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:37:11.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135874 2024-09-13T14:37:11.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135874 2024-09-13T14:37:11.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527 1-244813135874' 2024-09-13T14:37:11.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:11.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:37:11.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509469 2024-09-13T14:37:11.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509469 2024-09-13T14:37:11.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527 1-244813135874 2-64424509469' 2024-09-13T14:37:11.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:11.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:37:11.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364806 2024-09-13T14:37:11.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364806 2024-09-13T14:37:11.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527 1-244813135874 2-64424509469 3-214748364806' 2024-09-13T14:37:11.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:11.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836527 2024-09-13T14:37:11.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:11.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:37:11.964 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836527 2024-09-13T14:37:11.964 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:11.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836527 2024-09-13T14:37:11.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-09-13T14:37:11.965 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836527 2024-09-13T14:37:11.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:37:12.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836526 -lt 21474836527 2024-09-13T14:37:12.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:37:13.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:37:13.279 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:37:13.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836527 -lt 21474836527 2024-09-13T14:37:13.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:13.596 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135874 2024-09-13T14:37:13.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:13.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:37:13.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135874 2024-09-13T14:37:13.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:13.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135874 2024-09-13T14:37:13.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-09-13T14:37:13.601 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 244813135874 2024-09-13T14:37:13.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:37:13.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135874 -lt 244813135874 2024-09-13T14:37:13.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:13.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:13.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509469 2024-09-13T14:37:13.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:37:13.955 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509469 2024-09-13T14:37:13.955 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:13.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509469 2024-09-13T14:37:13.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-09-13T14:37:13.957 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509469 2024-09-13T14:37:13.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:37:14.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509469 -lt 64424509469 2024-09-13T14:37:14.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:14.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-214748364806 2024-09-13T14:37:14.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:14.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:37:14.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-214748364806 2024-09-13T14:37:14.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:14.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364806 2024-09-13T14:37:14.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 214748364806' 2024-09-13T14:37:14.301 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 214748364806 2024-09-13T14:37:14.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:37:14.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364806 -lt 214748364806 2024-09-13T14:37:14.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:37:14.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:37:14.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:37:15.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:37:15.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:37:15.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:37:15.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:37:15.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:37:15.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:37:15.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:37:15.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:37:15.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:37:15.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:37:15.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:37:15.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:37:15.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:37:15.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:37:15.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:37:15.783 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-13T14:37:15.783 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:37:15.783 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:37:15.783 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:37:15.783 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:37:16.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-13T14:37:16.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-13T14:37:16.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:37:16.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:37:16.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:37:16.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:37:16.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:37:16.110 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:37:16.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:36:43.237087+0000 2024-09-13T14:37:16.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:37:16.692 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:37:16.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:36:43.237087+0000 2024-09-13T14:37:16.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:37:16.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:36:43.237087+0000 2024-09-13T14:37:16.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:37:16.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:37:16.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:37:16.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:37:16.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:37:16.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:37:16.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:37:16.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:37:17.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:36:43.237087+0000 '>' 2024-09-13T14:36:43.237087+0000 2024-09-13T14:37:17.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:37:18.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:37:18.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:37:18.027 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:37:18.027 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:37:18.027 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:37:18.027 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:37:18.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:37:18.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:36:43.237087+0000 '>' 2024-09-13T14:36:43.237087+0000 2024-09-13T14:37:18.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:37:19.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:37:19.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:37:19.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:37:19.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:37:19.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:37:19.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:37:19.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:37:19.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:36:43.237087+0000 '>' 2024-09-13T14:36:43.237087+0000 2024-09-13T14:37:19.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:37:20.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:37:20.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:37:20.691 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:37:20.691 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:37:20.691 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:37:20.692 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:37:20.692 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:37:21.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:37:17.659467+0000 '>' 2024-09-13T14:36:43.237087+0000 2024-09-13T14:37:21.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:37:21.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:37:23.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:37:23.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:37:23.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:37:23.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:37:23.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:37:23.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:37:23.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:37:23.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:37:23.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:37:23.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:37:23.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:37:23.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:37:23.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:37:23.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:37:23.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:37:23.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:37:23.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:37:23.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:37:23.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:37:23.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:37:23.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:37:23.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:37:23.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:37:23.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-13T14:37:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-13T14:37:23.867 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:37:23.867 INFO:tasks.workunit.client.0.smithi112.stdout:hinfo_key 2024-09-13T14:37:23.867 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:37:24.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:37:24.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:37:24.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:37:24.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:37:24.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:37:24.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:37:24.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:37:24.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:37:24.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:37:24.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:37:24.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:37:24.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:37:24.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:37:24.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:37:24.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:37:24.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:37:24.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:37:24.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:37:24.160 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:37:24.160 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:37:24.160 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:37:24.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:37:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:37:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:37:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:37:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:37:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:37:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:37:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:37:24.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:37:24.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:37:24.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:37:24.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:37:24.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:37:24.163 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.3 2024-09-13T14:37:24.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:37:24.163 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:37:24.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:37:24.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:37:24.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:37:24.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:37:24.197 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:37:24.196+0000 7f86032787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:37:24.282 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:37:24.284+0000 7f86032787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:37:24.284 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:37:24.284+0000 7f86032787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:37:24.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:37:24.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:37:24.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:37:24.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:37:24.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:37:24.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:24.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:37:24.511 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:37:24.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:24.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:37:24.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:25.545 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:37:25.543+0000 7f86032787c0 -1 Falling back to public interface 2024-09-13T14:37:25.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:25.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:25.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:37:25.839 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:37:25.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:25.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:37:26.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:27.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:27.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:27.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:37:27.163 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:37:27.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:27.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:37:27.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:28.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:28.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:28.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:37:28.503 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:37:28.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:28.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:37:28.607 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:37:28.607+0000 7f86032787c0 -1 osd.3 59 log_to_monitors true 2024-09-13T14:37:28.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:29.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:29.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:29.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:37:29.842 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:37:29.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:29.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:37:30.223 INFO:tasks.workunit.client.0.smithi112.stderr:osd.3 up in weight 1 up_from 64 up_thru 64 down_at 60 last_clean_interval [50,59) [v2:127.0.0.1:6826/4154658820,v1:127.0.0.1:6827/4154658820] [v2:127.0.0.1:6828/4154658820,v1:127.0.0.1:6829/4154658820] exists,up 0c26729b-baf1-4c78-9d61-9982d2d70dc2 2024-09-13T14:37:30.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:37:30.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:37:30.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:37:30.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:37:30.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:37:30.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:37:30.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:37:30.223 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:37:30.224 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:37:30.224 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:37:30.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:37:30.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:37:30.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:37:30.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:37:30.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:37:30.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:37:30.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:37:30.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:37:30.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:37:30.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:37:30.677 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:37:30.677 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:37:30.677 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T14:37:30.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:37:30.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:30.678 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:37:30.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836532 2024-09-13T14:37:30.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836532 2024-09-13T14:37:30.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836532' 2024-09-13T14:37:30.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:30.840 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:37:31.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135879 2024-09-13T14:37:31.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135879 2024-09-13T14:37:31.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836532 1-244813135879' 2024-09-13T14:37:31.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:31.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:37:31.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509473 2024-09-13T14:37:31.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509473 2024-09-13T14:37:31.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836532 1-244813135879 2-64424509473' 2024-09-13T14:37:31.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:31.265 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:37:31.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906946 2024-09-13T14:37:31.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906946 2024-09-13T14:37:31.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836532 1-244813135879 2-64424509473 3-274877906946' 2024-09-13T14:37:31.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:31.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836532 2024-09-13T14:37:31.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:31.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:37:31.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836532 2024-09-13T14:37:31.410 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:31.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836532 2024-09-13T14:37:31.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2024-09-13T14:37:31.411 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836532 2024-09-13T14:37:31.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:37:31.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836531 -lt 21474836532 2024-09-13T14:37:31.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:37:32.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:37:32.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:37:33.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836532 -lt 21474836532 2024-09-13T14:37:33.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:33.059 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135879 2024-09-13T14:37:33.059 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:33.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:37:33.061 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135879 2024-09-13T14:37:33.061 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:33.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135879 2024-09-13T14:37:33.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135879' 2024-09-13T14:37:33.063 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 244813135879 2024-09-13T14:37:33.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:37:33.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135879 -lt 244813135879 2024-09-13T14:37:33.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:33.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509473 2024-09-13T14:37:33.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:33.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:37:33.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509473 2024-09-13T14:37:33.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:33.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509473 2024-09-13T14:37:33.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2024-09-13T14:37:33.401 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509473 2024-09-13T14:37:33.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:37:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509474 -lt 64424509473 2024-09-13T14:37:33.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:33.768 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906946 2024-09-13T14:37:33.769 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:33.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:37:33.771 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906946 2024-09-13T14:37:33.772 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:33.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906946 2024-09-13T14:37:33.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906946' 2024-09-13T14:37:33.773 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 274877906946 2024-09-13T14:37:33.773 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:37:34.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906946 -lt 274877906946 2024-09-13T14:37:34.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:37:34.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:37:34.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:37:34.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:37:34.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:37:34.502 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:37:34.502 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:37:34.502 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:37:34.502 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:37:34.503 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:37:34.503 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:37:34.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:37:34.837 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:37:34.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:37:34.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:37:35.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:37:35.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:37:35.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:37:35.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:37:35.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:37:35.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-09-13T14:37:35.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-13T14:37:35.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-13T14:37:35.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-09-13T14:37:35.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-09-13T14:37:35.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-13T14:37:35.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:37:35.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:37:35.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:37:35.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 183214"' 2024-09-13T14:37:35.295 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 183214' 2024-09-13T14:37:35.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T14:37:35.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:37:35.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:37:35.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 183216"' 2024-09-13T14:37:35.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:37:35.295 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 183216' 2024-09-13T14:37:35.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-13T14:37:35.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/183217: /' 2024-09-13T14:37:35.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 183214 183216' 2024-09-13T14:37:35.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:37:35.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:37:35.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 183214 2024-09-13T14:37:35.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T14:37:35.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/183219: /' 2024-09-13T14:37:38.621 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:37:38.621 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:37:38.621 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:37:38.621 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:37:38.621 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:37:38.621 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:37:38.621 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:37:38.621 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:37:38.621 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:37:38.621 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:37:38.622 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:37:38.622 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:37:38.622 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:37:38.622 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:37:38.622 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:37:38.622 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:37:38.622 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:37:38.622 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:37:38.622 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:37:38.622 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:37:38.622 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:37:38.622 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:37:38.622 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-13T14:37:38.623 INFO:tasks.workunit.client.0.smithi112.stderr:183217: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-13T14:37:38.623 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T14:37:38.623 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:37:38.623 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:37:38.623 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:37:38.623 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:37:38.623 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:37:38.623 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:37:38.623 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:37:38.626 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:37:38.626 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:37:38.626 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:37:38.626 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:37:38.626 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:37:38.626 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:37:38.627 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:37:38.627 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:37:38.627 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:37:38.627 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:37:38.627 INFO:tasks.workunit.client.0.smithi112.stderr:183217: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:37:38.627 INFO:tasks.workunit.client.0.smithi112.stderr:183217: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:37:38.627 INFO:tasks.workunit.client.0.smithi112.stderr:183217: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:37:38.627 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:37:38.627 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:37:38.627 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:37:38.627 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:37:38.627 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:37:38.627 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:37:38.628 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:37:38.628 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:37:38.628 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:37:38.628 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:37:38.628 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:37:38.628 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:37:38.628 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:37:38.628 INFO:tasks.workunit.client.0.smithi112.stderr:183217: start osd.1 2024-09-13T14:37:38.628 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:37:38.920 INFO:tasks.workunit.client.0.smithi112.stderr:1183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:37:38.920 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:37:38.920 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-13T14:37:38.920 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:37:38.920 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T14:37:38.921 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:37:38.921 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:37:38.921 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-13T14:37:38.921 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:37:38.921 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-13T14:37:38.921 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:37:38.921 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:37:38.921 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:37:38.921 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:37:38.921 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:37:38.921 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:37:38.921 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T14:37:38.921 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:37:38.922 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:37:38.922 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-13T14:37:38.922 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:37:38.922 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:37:38.922 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-13T14:37:38.922 INFO:tasks.workunit.client.0.smithi112.stderr:183219: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-13T14:37:38.922 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-13T14:37:38.922 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:37:38.922 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:37:38.922 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T14:37:38.922 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:37:38.922 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:37:38.923 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:37:38.923 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:37:38.925 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:37:38.925 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:37:38.925 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T14:37:38.925 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T14:37:38.925 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:37:38.925 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:37:38.925 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:37:38.926 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:37:38.926 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:37:38.926 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:37:38.926 INFO:tasks.workunit.client.0.smithi112.stderr:183219: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:37:38.926 INFO:tasks.workunit.client.0.smithi112.stderr:183219: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:37:38.926 INFO:tasks.workunit.client.0.smithi112.stderr:183219: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:37:38.926 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:37:38.926 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:37:38.926 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:37:38.926 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:37:38.926 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:37:38.926 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:37:38.926 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:37:38.927 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:37:38.927 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:37:38.927 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:37:38.927 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:37:38.927 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T14:37:38.927 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T14:37:38.927 INFO:tasks.workunit.client.0.smithi112.stderr:183219: start osd.2 2024-09-13T14:37:38.927 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:37:43.462 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:37:43.462 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:37:43.462 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:37:43.462 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:37:43.463 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:37:43.463 INFO:tasks.workunit.client.0.smithi112.stderr:183217: 2024-09-13T14:37:38.659+0000 7f40a86e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:37:43.463 INFO:tasks.workunit.client.0.smithi112.stderr:183217: 2024-09-13T14:37:38.663+0000 7f40a86e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:37:43.463 INFO:tasks.workunit.client.0.smithi112.stderr:183217: 2024-09-13T14:37:38.663+0000 7f40a86e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:37:43.463 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:37:43.463 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:37:43.463 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:37:43.463 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:37:43.463 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:37:43.463 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:43.463 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:37:43.463 INFO:tasks.workunit.client.0.smithi112.stderr:183217: 0 2024-09-13T14:37:43.463 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:43.464 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:37:43.464 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:43.464 INFO:tasks.workunit.client.0.smithi112.stderr:183217: 2024-09-13T14:37:40.295+0000 7f40a86e37c0 -1 Falling back to public interface 2024-09-13T14:37:43.464 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:43.464 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:43.464 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:37:43.464 INFO:tasks.workunit.client.0.smithi112.stderr:183217: 1 2024-09-13T14:37:43.464 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:43.464 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:37:43.464 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:43.464 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:43.464 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:43.464 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:37:43.465 INFO:tasks.workunit.client.0.smithi112.stderr:183217: 2 2024-09-13T14:37:43.465 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:43.465 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:37:43.465 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:43.465 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:43.465 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:43.465 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:37:43.465 INFO:tasks.workunit.client.0.smithi112.stderr:183217: 3 2024-09-13T14:37:43.465 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:43.465 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:37:43.465 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:44.003 INFO:tasks.workunit.client.0.smithi112.stderr:183217: 2024-09-13T14:37:43.459+0000 7f40a86e37c0 -83219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-13T14:37:44.003 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T14:37:44.003 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:37:44.003 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:37:44.003 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:37:44.003 INFO:tasks.workunit.client.0.smithi112.stderr:183219: 2024-09-13T14:37:38.955+0000 7f96a0e537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:37:44.003 INFO:tasks.workunit.client.0.smithi112.stderr:183219: 2024-09-13T14:37:38.959+0000 7f96a0e537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:37:44.003 INFO:tasks.workunit.client.0.smithi112.stderr:183219: 2024-09-13T14:37:38.959+0000 7f96a0e537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:37:44.004 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T14:37:44.004 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:37:44.004 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:37:44.004 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:37:44.004 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:37:44.004 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:44.004 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:37:44.004 INFO:tasks.workunit.client.0.smithi112.stderr:183219: 0 2024-09-13T14:37:44.004 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:44.004 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:37:44.004 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:44.005 INFO:tasks.workunit.client.0.smithi112.stderr:183219: 2024-09-13T14:37:40.467+0000 7f96a0e537c0 -1 Falling back to public interface 2024-09-13T14:37:44.005 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:44.005 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:44.005 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:37:44.005 INFO:tasks.workunit.client.0.smithi112.stderr:183219: 1 2024-09-13T14:37:44.005 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:44.005 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:37:44.005 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:44.005 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:44.005 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:44.005 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:37:44.005 INFO:tasks.workunit.client.0.smithi112.stderr:183219: 2 2024-09-13T14:37:44.005 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:44.006 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:37:44.006 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:44.006 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:44.006 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:44.006 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:37:44.006 INFO:tasks.workunit.client.0.smithi112.stderr:183219: 3 2024-09-13T14:37:44.006 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:44.006 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:37:44.006 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:37:45.765 INFO:tasks.workunit.client.0.smithi112.stderr:183219: 2024-09-13T14:37:44.003+0000 7f96a0e537c0 -1 osd.1 66 log_to_monitors true 2024-09-13T14:37:45.765 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:45.765 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:45.765 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:37:45.765 INFO:tasks.workunit.client.0.smithi112.stderr:183217: 4 2024-09-13T14:37:45.765 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:45.765 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:37:45.765 INFO:tasks.workunit.client.0.smithi112.stderr:183217: osd.1 up in weight 1 up_from 72 up_thru 60 down_at 67 last_clean_interval [57,66) [v2:127.0.0.1:6810/283447613,v1:127.0.0.1:6811/283447613] [v2:127.0.0.1:6812/283447613,v1:127.0.0.1:6813/283447613] exists,up 6bbc9a9f-3432-44d8-a504-eec0cac352ac 2024-09-13T14:37:45.765 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:37:45.766 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:37:45.766 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:37:45.766 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:37:45.766 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:37:45.766 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:37:45.766 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:37:45.766 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:37:45.766 INFO:tasks.workunit.client.0.smithi112.stderr:183217: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:37:45.766 INFO:tasks.workunit.client.0.smithi112.stderr:183217: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:37:45.766 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:37:45.766 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:37:45.766 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:37:45.766 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:37:45.767 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:37:45.767 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:37:45.767 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:37:45.767 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:37:45.767 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:37:45.767 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:37:45.767 INFO:tasks.workunit.client.0.smithi112.stderr:183217: 1 2024-09-13T14:37:45.767 INFO:tasks.workunit.client.0.smithi112.stderr:183217: 2 2024-09-13T14:37:45.767 INFO:tasks.workunit.client.0.smithi112.stderr:183217: 3' 2024-09-13T14:37:45.767 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:37:45.768 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:45.768 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:37:45.768 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836536 2024-09-13T14:37:45.768 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836536 2024-09-13T14:37:45.768 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536' 2024-09-13T14:37:45.768 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:45.966 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.1 osd.2 66 log_to_monitors true 2024-09-13T14:37:45.966 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:37:45.966 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:37:45.966 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:37:45.966 INFO:tasks.workunit.client.0.smithi112.stderr:183219: 4 2024-09-13T14:37:45.966 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:37:45.966 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:37:45.966 INFO:tasks.workunit.client.0.smithi112.stderr:183219: osd.2 up in weight 1 up_from 73 up_thru 53 down_at 67 last_clean_interval [15,66) [v2:127.0.0.1:6818/750891439,v1:127.0.0.1:6819/750891439] [v2:127.0.0.1:6820/750891439,v1:127.0.0.1:6821/750891439] exists,up c38afcf9-9793-45de-9f1c-b2f5c8d69cc9 2024-09-13T14:37:45.967 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:37:45.967 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:37:45.967 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:37:45.967 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:37:45.967 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:37:45.967 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:37:45.967 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:37:45.967 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:37:45.967 INFO:tasks.workunit.client.0.smithi112.stderr:183219: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:37:45.967 INFO:tasks.workunit.client.0.smithi112.stderr:183219: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:37:45.967 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:37:45.967 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:37:45.967 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:37:45.967 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:37:45.968 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:37:45.968 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:37:45.968 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:37:45.968 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:37:45.968 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:37:45.968 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:37:45.968 INFO:tasks.workunit.client.0.smithi112.stderr:183219: 1 2024-09-13T14:37:45.968 INFO:tasks.workunit.client.0.smithi112.stderr:183219: 2 2024-09-13T14:37:45.968 INFO:tasks.workunit.client.0.smithi112.stderr:183219: 3' 2024-09-13T14:37:45.968 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:37:45.968 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:45.968 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:37:45.968 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836537 2024-09-13T14:37:45.968 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836537 2024-09-13T14:37:45.969 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537' 2024-09-13T14:37:45.969 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:48.026 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:37:48.026 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645314 2024-09-13T14:37:48.026 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645314 2024-09-13T14:37:48.026 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-309237645314' 2024-09-13T14:37:48.026 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:48.026 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:37:48.026 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612610 2024-09-13T14:37:48.026 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612610 2024-09-13T14:37:48.027 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-309237645314 2-313532612610' 2024-09-13T14:37:48.027 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:48.027 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:37:48.027 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906950 2024-09-13T14:37:48.027 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906950 2024-09-13T14:37:48.027 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-309237645314 2-313532612610 3-274877906950' 2024-09-13T14:37:48.027 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:48.027 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836536 2024-09-13T14:37:48.027 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:48.027 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:37:48.027 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836536 2024-09-13T14:37:48.027 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:48.027 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836536 2024-09-13T14:37:48.027 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-09-13T14:37:48.028 INFO:tasks.workunit.client.0.smithi112.stderr:183217: waiting osd.0 seq 21474836536 2024-09-13T14:37:48.028 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:37:48.028 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836535 -lt 21474836536 2024-09-13T14:37:48.028 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:37:48.028 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:37:48.028 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:37:48.028 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836537 -lt 21474836536 2024-09-13T14:37:48.028 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:48.028 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-309237645314 2024-09-13T14:37:48.028 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:48.028 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:37:48.028 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-309237645314 2024-09-13T14:37:48.029 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:48.172 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntush:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:37:48.172 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645315 2024-09-13T14:37:48.172 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645315 2024-09-13T14:37:48.172 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-309237645315' 2024-09-13T14:37:48.172 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:48.172 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:37:48.172 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612611 2024-09-13T14:37:48.172 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612611 2024-09-13T14:37:48.172 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-309237645315 2-313532612611' 2024-09-13T14:37:48.172 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:37:48.173 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:37:48.173 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906951 2024-09-13T14:37:48.173 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906951 2024-09-13T14:37:48.173 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-309237645315 2-313532612611 3-274877906951' 2024-09-13T14:37:48.173 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:48.173 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836537 2024-09-13T14:37:48.173 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:48.173 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:37:48.173 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836537 2024-09-13T14:37:48.173 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:48.173 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836537 2024-09-13T14:37:48.173 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-09-13T14:37:48.173 INFO:tasks.workunit.client.0.smithi112.stderr:183219: waiting osd.0 seq 21474836537 2024-09-13T14:37:48.174 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:37:48.174 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836535 -lt 21474836537 2024-09-13T14:37:48.174 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:37:48.174 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:37:48.174 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:37:48.174 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836537 -lt 21474836537 2024-09-13T14:37:48.174 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:48.174 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-309237645315 2024-09-13T14:37:48.174 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:48.174 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:37:48.174 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-309237645315 2024-09-13T14:37:48.174 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:49.425 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645314 2024-09-13T14:37:49.425 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 309237645314' 2024-09-13T14:37:49.425 INFO:tasks.workunit.client.0.smithi112.stderr:183217: waiting osd.1 seq 309237645314 2024-09-13T14:37:49.425 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:37:49.425 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645315 -lt 309237645314 2024-09-13T14:37:49.425 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:49.426 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-313532612610 2024-09-13T14:37:49.426 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:49.426 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:37:49.426 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-313532612610 2024-09-13T14:37:49.426 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:49.426 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612610 2024-09-13T14:37:49.426 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-09-13T14:37:49.426 INFO:tasks.workunit.client.0.smithi112.stderr:183217: waiting osd.2 seq 313532612610 2024-09-13T14:37:49.426 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:37:49.426 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612611 -lt 313532612610 2024-09-13T14:37:49.426 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:49.426 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906950 2024-09-13T14:37:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:37:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906950 2024-09-13T14:37:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906950 2024-09-13T14:37:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906950' 2024-09-13T14:37:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:183217: waiting osd.3 seq 274877906950 2024-09-13T14:37:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:37:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906951 -lt 274877906950 2024-09-13T14:37:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:37:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:37:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:37:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:37:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:37:49.428 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:37:49.428 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:37:49.428 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:37:49.586 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645315 2024-09-13T14:37:49.587 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 309237645315' 2024-09-13T14:37:49.587 INFO:tasks.workunit.client.0.smithi112.stderr:183219: waiting osd.1 seq 309237645315 2024-09-13T14:37:49.587 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:37:49.587 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645315 -lt 309237645315 2024-09-13T14:37:49.587 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:49.587 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-313532612611 2024-09-13T14:37:49.587 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:49.587 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:37:49.587 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-313532612611 2024-09-13T14:37:49.587 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:49.587 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612611 2024-09-13T14:37:49.587 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 313532612611' 2024-09-13T14:37:49.588 INFO:tasks.workunit.client.0.smithi112.stderr:183219: waiting osd.2 seq 313532612611 2024-09-13T14:37:49.588 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:37:49.588 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612611 -lt 313532612611 2024-09-13T14:37:49.588 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:37:49.588 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906951 2024-09-13T14:37:49.588 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:37:49.588 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:37:49.588 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906951 2024-09-13T14:37:49.588 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:37:49.588 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906951 2024-09-13T14:37:49.588 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906951' 2024-09-13T14:37:49.588 INFO:tasks.workunit.client.0.smithi112.stderr:183219: waiting osd.3 seq 274877906951 2024-09-13T14:37:49.588 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:37:49.588 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906951 -lt 274877906951 2024-09-13T14:37:49.589 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:37:49.589 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:37:49.589 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:37:49.589 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:37:49.589 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:37:49.589 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:37:49.589 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:37:49.589 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:37:50.998 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:37:50.998 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:37:50.998 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:37:50.998 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T14:37:50.998 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:37:50.998 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:37:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:37:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-13T14:37:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-13T14:37:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T14:37:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-13T14:37:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:37:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:37:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:37:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:37:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:37:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:37:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:37:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:37:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:37:51.000 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:37:51.000 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T14:37:51.000 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:37:51.000 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:37:51.000 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:37:51.000 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-13T14:37:51.000 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-13T14:37:51.000 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T14:37:51.000 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T14:37:51.000 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T14:37:51.000 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T14:37:51.000 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T14:37:51.074 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home") | not)' 2024-09-13T14:37:51.074 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:37:51.074 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:37:51.074 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T14:37:51.074 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:37:51.074 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:37:51.074 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:37:51.074 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-13T14:37:51.074 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-13T14:37:51.074 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T14:37:51.075 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-13T14:37:51.075 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:37:51.075 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:37:51.075 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:37:51.075 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:37:51.075 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:37:51.075 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:37:51.075 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:37:51.075 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:37:51.075 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:37:51.075 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:37:51.075 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T14:37:51.076 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:37:51.076 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:37:51.076 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:37:51.076 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-13T14:37:51.076 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-13T14:37:51.076 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T14:37:51.076 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T14:37:51.076 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T14:37:51.076 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T14:37:51.076 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T14:37:52.332 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T14:37:52.332 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T14:37:52.332 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T14:37:52.333 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T14:37:52.333 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T14:37:52.333 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:37:52.333 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T14:37:52.333 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:37:52.333 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:37:52.333 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:37:52.333 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:37:52.333 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:37:52.333 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:37:52.333 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:37:52.333 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:37:52.334 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:37:52.334 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:37:52.334 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:37:52.334 INFO:tasks.workunit.client.0.smithi112.stderr:183217: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:37:52.334 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:37:52.334 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:37:52.334 INFO:tasks.workunit.client.0.smithi112.stderr:183217: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:37:52.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:37:52.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:37:52.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 183216 2024-09-13T14:37:52.423 INFO:tasks.workunit.client.0.smithi112.stderr:/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T14:37:52.423 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T14:37:52.423 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T14:37:52.423 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T14:37:52.423 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T14:37:52.423 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:37:52.423 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T14:37:52.423 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:37:52.423 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:37:52.423 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:37:52.424 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:37:52.424 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:37:52.424 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:37:52.424 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:37:52.424 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:37:52.424 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:37:52.424 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:37:52.424 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:37:52.424 INFO:tasks.workunit.client.0.smithi112.stderr:183219: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:37:52.424 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:37:52.424 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:37:52.424 INFO:tasks.workunit.client.0.smithi112.stderr:183219: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:37:52.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:37:52.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:37:52.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:37:52.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:37:52.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-13T14:37:52.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:37:52.426 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-13T14:37:52.426 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:37:52.426 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:37:52.426 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:37:52.426 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:37:52.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-13T14:37:52.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-13T14:37:52.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:37:52.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:37:52.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:37:52.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:37:52.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:37:52.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:37:53.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:37:17.659467+0000 2024-09-13T14:37:53.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:37:53.406 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:37:53.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:37:17.659467+0000 2024-09-13T14:37:53.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:37:53.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:37:17.659467+0000 2024-09-13T14:37:53.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:37:53.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:37:53.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:37:53.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:37:53.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:37:53.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:37:53.424 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:37:53.424 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:37:53.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:37:17.659467+0000 '>' 2024-09-13T14:37:17.659467+0000 2024-09-13T14:37:53.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:37:54.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:37:54.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:37:54.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:37:54.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:37:54.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:37:54.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:37:54.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:37:55.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:37:53.662789+0000 '>' 2024-09-13T14:37:17.659467+0000 2024-09-13T14:37:55.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:37:55.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:37:57.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-13T14:37:57.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:37:57.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:37:57.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:37:57.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 186304"' 2024-09-13T14:37:57.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 186304' 2024-09-13T14:37:57.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-13T14:37:57.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:37:57.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:37:57.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 186305"' 2024-09-13T14:37:57.075 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 186305' 2024-09-13T14:37:57.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-13T14:37:57.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 186304 186305' 2024-09-13T14:37:57.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:37:57.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:37:57.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 186304 2024-09-13T14:37:57.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:37:57.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-13T14:37:57.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/186309: /' 2024-09-13T14:37:57.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/186307: /' 2024-09-13T14:37:58.393 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:37:58.393 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:37:58.393 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:37:58.393 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:37:58.393 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:37:58.393 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:37:58.393 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:37:58.393 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:37:58.393 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:37:58.393 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:37:58.393 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:37:58.394 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:37:58.394 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:37:58.394 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:37:58.394 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:37:58.394 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:37:58.394 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:37:58.394 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:37:58.394 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:37:58.394 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:37:58.394 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:37:58.394 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:37:58.394 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-13T14:37:58.394 INFO:tasks.workunit.client.0.smithi112.stderr:186307: _ 2024-09-13T14:37:58.395 INFO:tasks.workunit.client.0.smithi112.stderr:186307: hinfo_key 2024-09-13T14:37:58.395 INFO:tasks.workunit.client.0.smithi112.stderr:186307: snapset 2024-09-13T14:37:58.395 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T14:37:58.395 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:37:58.395 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:37:58.395 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:37:58.395 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:37:58.395 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:37:58.395 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:37:58.395 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:37:58.398 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:37:58.398 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:37:58.398 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:37:58.399 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:37:58.399 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:37:58.399 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:37:58.399 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:37:58.399 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:37:58.399 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:37:58.399 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:37:58.399 INFO:tasks.workunit.client.0.smithi112.stderr:186307: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:37:58.399 INFO:tasks.workunit.client.0.smithi112.stderr:186307: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:37:58.399 INFO:tasks.workunit.client.0.smithi112.stderr:186307: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:37:58.399 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:37:58.399 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:37:58.399 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:37:58.400 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:37:58.400 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:37:58.400 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:37:58.400 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:37:58.400 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:37:58.400 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:37:58.400 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:37:58.400 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:37:58.400 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:37:58.400 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:37:58.400 INFO:tasks.workunit.client.0.smithi112.stderr:186307: start osd.1 2024-09-13T14:37:58.412 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:37:58.413 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:37:58.413 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-13T14:37:58.413 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:37:58.413 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-13T14:37:58.413 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:37:58.413 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:37:58.413 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-13T14:37:58.413 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:37:58.413 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-13T14:37:58.413 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:37:58.413 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:37:58.413 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:37:58.413 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:37:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:37:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:37:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-13T14:37:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:37:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:37:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-13T14:37:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:37:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:37:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-09-13T14:37:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:186309: _ 2024-09-13T14:37:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:186309: hinfo_key 2024-09-13T14:37:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:186309: snapset 2024-09-13T14:37:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-13T14:37:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:37:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:37:58.415 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T14:37:58.415 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:37:58.415 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:37:58.415 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:37:58.415 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:37:58.416 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:37:58.416 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:37:58.416 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T14:37:58.416 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T14:37:58.416 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:37:58.416 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:37:58.416 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:37:58.416 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:37:58.416 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:37:58.417 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:37:58.417 INFO:tasks.workunit.client.0.smithi112.stderr:186309: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:37:58.417 INFO:tasks.workunit.client.0.smithi112.stderr:186309: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:37:58.417 INFO:tasks.workunit.client.0.smithi112.stderr:186309: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:37:58.417 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:37:58.417 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:37:58.417 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:37:58.417 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:37:58.417 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:37:58.417 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:37:58.417 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:37:58.417 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:37:58.417 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:37:58.418 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:37:58.418 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:37:58.418 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T14:37:58.418 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T14:37:58.418 INFO:tasks.workunit.client.0.smithi112.stderr:186309: start osd.2 2024-09-13T14:38:03.263 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:38:03.264 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:38:03.264 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:38:03.264 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:38:03.264 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:38:03.264 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:38:03.264 INFO:tasks.workunit.client.0.smithi112.stderr:186307: 2024-09-13T14:37:58.435+0000 7f13661887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:03.264 INFO:tasks.workunit.client.0.smithi112.stderr:186307: 2024-09-13T14:37:58.531+0000 7f13661887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:03.264 INFO:tasks.workunit.client.0.smithi112.stderr:186307: 2024-09-13T14:37:58.531+0000 7f13661887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:03.264 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:38:03.264 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:38:03.265 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:38:03.265 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:38:03.265 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:38:03.265 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:03.265 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:38:03.265 INFO:tasks.workunit.client.0.smithi112.stderr:186307: 0 2024-09-13T14:38:03.265 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:03.265 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:03.265 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:03.265 INFO:tasks.workunit.client.0.smithi112.stderr:186307: 2024-09-13T14:37:59.831+0000 7f13661887c0 -1 Falling back to public interface 2024-09-13T14:38:03.265 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:03.265 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:03.265 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:38:03.266 INFO:tasks.workunit.client.0.smithi112.stderr:186307: 1 2024-09-13T14:38:03.266 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:03.266 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:03.266 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:03.266 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:03.266 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:03.266 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:38:03.266 INFO:tasks.workunit.client.0.smithi112.stderr:186307: 2 2024-09-13T14:38:03.266 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:03.266 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:03.266 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:03.266 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:03.266 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:03.266 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:38:03.267 INFO:tasks.workunit.client.0.smithi112.stderr:186307: 3 2024-09-13T14:38:03.267 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:03.267 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:03.267 INFO:tasks.workunit.client.0.smithi112.stderr:186307: 2024-09-13T14:38:03.166+0000 7f13661887c0 -1 osd.1 75 log_to_monitors true 2024-09-13T14:38:04.268 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cevery_ops 2024-09-13T14:38:04.268 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-13T14:38:04.268 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T14:38:04.269 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:38:04.269 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:38:04.269 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:38:04.269 INFO:tasks.workunit.client.0.smithi112.stderr:186309: 2024-09-13T14:37:58.447+0000 7f976eaf87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:04.269 INFO:tasks.workunit.client.0.smithi112.stderr:186309: 2024-09-13T14:37:58.531+0000 7f976eaf87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:04.269 INFO:tasks.workunit.client.0.smithi112.stderr:186309: 2024-09-13T14:37:58.531+0000 7f976eaf87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:04.269 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T14:38:04.269 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:38:04.269 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:38:04.269 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:38:04.269 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:38:04.269 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:04.269 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:38:04.270 INFO:tasks.workunit.client.0.smithi112.stderr:186309: 0 2024-09-13T14:38:04.270 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:04.270 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:38:04.270 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:04.270 INFO:tasks.workunit.client.0.smithi112.stderr:186309: 2024-09-13T14:38:00.051+0000 7f976eaf87c0 -1 Falling back to public interface 2024-09-13T14:38:04.270 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:04.271 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:04.271 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:38:04.271 INFO:tasks.workunit.client.0.smithi112.stderr:186309: 1 2024-09-13T14:38:04.271 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:04.271 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:38:04.271 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:04.271 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:04.271 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:04.271 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:38:04.271 INFO:tasks.workunit.client.0.smithi112.stderr:186309: 2 2024-09-13T14:38:04.271 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:04.271 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:38:04.271 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:04.271 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:04.271 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:04.272 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:38:04.272 INFO:tasks.workunit.client.0.smithi112.stderr:186309: 3 2024-09-13T14:38:04.272 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:04.272 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:38:04.272 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:05.388 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.clienph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:05.388 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:05.388 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:05.388 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:38:05.388 INFO:tasks.workunit.client.0.smithi112.stderr:186307: 4 2024-09-13T14:38:05.388 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:05.389 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:05.389 INFO:tasks.workunit.client.0.smithi112.stderr:186307: osd.1 up in weight 1 up_from 80 up_thru 80 down_at 76 last_clean_interval [72,75) [v2:127.0.0.1:6810/992739458,v1:127.0.0.1:6811/992739458] [v2:127.0.0.1:6812/992739458,v1:127.0.0.1:6813/992739458] exists,up 6bbc9a9f-3432-44d8-a504-eec0cac352ac 2024-09-13T14:38:05.389 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:38:05.389 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:38:05.389 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:38:05.389 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:38:05.389 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:38:05.389 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:38:05.389 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:38:05.389 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:38:05.389 INFO:tasks.workunit.client.0.smithi112.stderr:186307: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:38:05.389 INFO:tasks.workunit.client.0.smithi112.stderr:186307: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:38:05.389 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:38:05.390 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:38:05.390 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:38:05.390 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:38:05.390 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:38:05.390 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:38:05.390 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:38:05.390 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:38:05.390 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:38:05.390 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:38:05.390 INFO:tasks.workunit.client.0.smithi112.stderr:186307: 1 2024-09-13T14:38:05.390 INFO:tasks.workunit.client.0.smithi112.stderr:186307: 2 2024-09-13T14:38:05.390 INFO:tasks.workunit.client.0.smithi112.stderr:186307: 3' 2024-09-13T14:38:05.390 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:38:05.391 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:05.391 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:38:05.391 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836542 2024-09-13T14:38:05.391 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836542 2024-09-13T14:38:05.391 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542' 2024-09-13T14:38:05.391 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:06.667 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/cept.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:06.668 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:06.668 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:38:06.668 INFO:tasks.workunit.client.0.smithi112.stderr:186309: 4 2024-09-13T14:38:06.668 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:06.668 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:38:06.668 INFO:tasks.workunit.client.0.smithi112.stderr:186309: 2024-09-13T14:38:04.538+0000 7f976eaf87c0 -1 osd.2 75 log_to_monitors true 2024-09-13T14:38:06.668 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:06.668 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:06.668 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:06.668 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:38:06.668 INFO:tasks.workunit.client.0.smithi112.stderr:186309: 5 2024-09-13T14:38:06.668 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:38:06.668 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:06.669 INFO:tasks.workunit.client.0.smithi112.stderr:186309: osd.2 up in weight 1 up_from 84 up_thru 53 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6818/589205765,v1:127.0.0.1:6819/589205765] [v2:127.0.0.1:6820/589205765,v1:127.0.0.1:6821/589205765] exists,up c38afcf9-9793-45de-9f1c-b2f5c8d69cc9 2024-09-13T14:38:06.669 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:38:06.669 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:38:06.669 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:38:06.669 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:38:06.669 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:38:06.669 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:38:06.669 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:38:06.669 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:38:06.669 INFO:tasks.workunit.client.0.smithi112.stderr:186309: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:38:06.669 INFO:tasks.workunit.client.0.smithi112.stderr:186309: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:38:06.669 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:38:06.669 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:38:06.669 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:38:06.670 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:38:06.670 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:38:06.670 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:38:06.670 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:38:06.670 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:38:06.670 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:38:06.670 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:38:06.670 INFO:tasks.workunit.client.0.smithi112.stderr:186309: 1 2024-09-13T14:38:06.670 INFO:tasks.workunit.client.0.smithi112.stderr:186309: 2 2024-09-13T14:38:06.670 INFO:tasks.workunit.client.0.smithi112.stderr:186309: 3' 2024-09-13T14:38:06.670 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:38:06.670 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:07.603 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //h-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:38:07.603 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383682 2024-09-13T14:38:07.603 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383682 2024-09-13T14:38:07.603 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-343597383682' 2024-09-13T14:38:07.603 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:38:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:186307: Error ENXIO: problem getting command descriptions from osd.2 2024-09-13T14:38:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-13T14:38:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-13T14:38:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-13T14:38:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:38:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906956 2024-09-13T14:38:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906956 2024-09-13T14:38:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-343597383682 3-274877906956' 2024-09-13T14:38:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836542 2024-09-13T14:38:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:38:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836542 2024-09-13T14:38:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836542 2024-09-13T14:38:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2024-09-13T14:38:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:186307: waiting osd.0 seq 21474836542 2024-09-13T14:38:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:38:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836541 -lt 21474836542 2024-09-13T14:38:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:38:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:38:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:38:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836542 -lt 21474836542 2024-09-13T14:38:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-343597383682 2024-09-13T14:38:07.605 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:07.606 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:38:07.606 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-343597383682 2024-09-13T14:38:07.606 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:09.101 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:38:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836543 2024-09-13T14:38:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836543 2024-09-13T14:38:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543' 2024-09-13T14:38:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:38:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383683 2024-09-13T14:38:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383683 2024-09-13T14:38:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-343597383683' 2024-09-13T14:38:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:38:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252866 2024-09-13T14:38:09.102 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252866 2024-09-13T14:38:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-343597383683 2-360777252866' 2024-09-13T14:38:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:38:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906957 2024-09-13T14:38:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906957 2024-09-13T14:38:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-343597383683 2-360777252866 3-274877906957' 2024-09-13T14:38:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836543 2024-09-13T14:38:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:38:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836543 2024-09-13T14:38:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836543 2024-09-13T14:38:09.103 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-09-13T14:38:09.104 INFO:tasks.workunit.client.0.smithi112.stderr:186309: waiting osd.0 seq 21474836543 2024-09-13T14:38:09.104 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:38:09.104 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836542 -lt 21474836543 2024-09-13T14:38:09.104 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:38:09.104 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:38:09.104 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:38:09.104 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836544 -lt 21474836543 2024-09-13T14:38:09.594 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.clientcephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383682 2024-09-13T14:38:09.595 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 343597383682' 2024-09-13T14:38:09.595 INFO:tasks.workunit.client.0.smithi112.stderr:186307: waiting osd.1 seq 343597383682 2024-09-13T14:38:09.595 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:38:09.595 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383682 -lt 343597383682 2024-09-13T14:38:09.595 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:09.595 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906956 2024-09-13T14:38:09.595 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:09.595 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:38:09.595 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906956 2024-09-13T14:38:09.596 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:09.596 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906956 2024-09-13T14:38:09.596 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906956' 2024-09-13T14:38:09.596 INFO:tasks.workunit.client.0.smithi112.stderr:186307: waiting osd.3 seq 274877906956 2024-09-13T14:38:09.596 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:38:09.596 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906956 -lt 274877906956 2024-09-13T14:38:09.596 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:38:09.596 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:09.596 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:09.596 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:38:09.596 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:38:09.596 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:38:09.596 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:38:09.597 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:38:09.597 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:38:09.597 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:38:09.597 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:38:09.597 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:38:09.597 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:38:09.597 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:09.597 INFO:tasks.workunit.client.0.smithi112.stderr:186307: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:09.597 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:38:09.597 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:38:09.597 INFO:tasks.workunit.client.0.smithi112.stderr:186307: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:38:09.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:38:09.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:38:09.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 186305 2024-09-13T14:38:10.566 INFO:tasks.workunit.client.0.smithi112.stderr:.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:10.566 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-343597383683 2024-09-13T14:38:10.566 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:10.566 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:38:10.566 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-343597383683 2024-09-13T14:38:10.566 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:10.566 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383683 2024-09-13T14:38:10.566 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 343597383683' 2024-09-13T14:38:10.566 INFO:tasks.workunit.client.0.smithi112.stderr:186309: waiting osd.1 seq 343597383683 2024-09-13T14:38:10.567 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:38:10.567 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383684 -lt 343597383683 2024-09-13T14:38:10.567 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:10.567 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-360777252866 2024-09-13T14:38:10.567 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:10.567 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:38:10.567 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-360777252866 2024-09-13T14:38:10.567 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:10.567 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252866 2024-09-13T14:38:10.567 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 360777252866' 2024-09-13T14:38:10.567 INFO:tasks.workunit.client.0.smithi112.stderr:186309: waiting osd.2 seq 360777252866 2024-09-13T14:38:10.567 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:38:10.567 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252866 -lt 360777252866 2024-09-13T14:38:10.567 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:10.568 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-274877906957 2024-09-13T14:38:10.568 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:10.568 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:38:10.568 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-274877906957 2024-09-13T14:38:10.568 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:10.568 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906957 2024-09-13T14:38:10.568 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 274877906957' 2024-09-13T14:38:10.568 INFO:tasks.workunit.client.0.smithi112.stderr:186309: waiting osd.3 seq 274877906957 2024-09-13T14:38:10.568 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:38:10.568 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906957 -lt 274877906957 2024-09-13T14:38:10.568 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:38:10.568 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:10.568 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:38:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:38:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:38:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:38:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:38:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:38:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:38:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:38:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:38:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:38:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:11.321 INFO:tasks.workunit.client.0.smithi112.stderr:186309: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:11.321 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:38:11.321 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:38:11.321 INFO:tasks.workunit.client.0.smithi112.stderr:186309: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:38:11.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:38:11.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:38:11.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:38:11.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:38:11.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-13T14:38:11.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:38:11.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:38:11.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:38:11.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-09-13T14:38:11.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-13T14:38:11.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-13T14:38:11.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-13T14:38:11.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-09-13T14:38:11.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-13T14:38:11.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:38:11.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:38:11.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:38:11.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 188680"' 2024-09-13T14:38:11.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 188680' 2024-09-13T14:38:11.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:38:11.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:38:11.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:38:11.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 188682"' 2024-09-13T14:38:11.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:38:11.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 188682' 2024-09-13T14:38:11.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-13T14:38:11.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 188680 188682' 2024-09-13T14:38:11.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:38:11.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:38:11.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 188680 2024-09-13T14:38:11.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/188683: /' 2024-09-13T14:38:11.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:38:11.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/188685: /' 2024-09-13T14:38:14.924 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:38:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:38:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:38:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:38:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:38:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:38:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:38:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:38:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:38:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:38:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:38:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:38:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:38:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:38:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:38:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:38:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:38:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:38:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:38:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:38:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:38:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:38:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-13T14:38:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:188683: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-13T14:38:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:38:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:38:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:38:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:38:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:38:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:38:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:38:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:38:14.929 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:38:14.929 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:38:14.929 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:38:14.929 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:38:14.929 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:38:14.929 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:38:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:38:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:38:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:38:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:38:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:188683: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:38:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:188683: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:38:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:188683: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:38:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:38:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:38:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:38:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:38:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:38:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:38:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:38:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:38:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:38:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:38:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:38:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:38:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:38:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:188683: start osd.3 2024-09-13T14:38:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:38:15.248 INFO:tasks.workunit.client.0.smithi112.stderr:1188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:38:15.249 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:38:15.249 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:38:15.249 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:38:15.249 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:38:15.249 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:38:15.249 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:38:15.249 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:38:15.249 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:38:15.249 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:38:15.249 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:38:15.249 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:38:15.249 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:38:15.249 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:38:15.250 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:38:15.250 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:38:15.250 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:38:15.250 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:38:15.250 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:38:15.250 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:38:15.250 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:38:15.250 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:38:15.250 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-13T14:38:15.250 INFO:tasks.workunit.client.0.smithi112.stderr:188685: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-13T14:38:15.250 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T14:38:15.250 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:38:15.250 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:38:15.251 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:38:15.251 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:38:15.251 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:38:15.251 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:38:15.251 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:38:15.253 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:38:15.253 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:38:15.253 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:38:15.253 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:38:15.253 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:38:15.253 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:38:15.253 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:38:15.253 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:38:15.254 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:38:15.254 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:38:15.254 INFO:tasks.workunit.client.0.smithi112.stderr:188685: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:38:15.254 INFO:tasks.workunit.client.0.smithi112.stderr:188685: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:38:15.254 INFO:tasks.workunit.client.0.smithi112.stderr:188685: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:38:15.254 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:38:15.254 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:38:15.254 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:38:15.254 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:38:15.254 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:38:15.254 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:38:15.254 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:38:15.254 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:38:15.254 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:38:15.255 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:38:15.255 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:38:15.255 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:38:15.255 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:38:15.255 INFO:tasks.workunit.client.0.smithi112.stderr:188685: start osd.1 2024-09-13T14:38:15.255 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:38:19.590 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:38:19.590 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:38:19.590 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:38:19.590 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:38:19.591 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:38:19.591 INFO:tasks.workunit.client.0.smithi112.stderr:188683: 2024-09-13T14:38:14.962+0000 7f31638667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:19.591 INFO:tasks.workunit.client.0.smithi112.stderr:188683: 2024-09-13T14:38:14.966+0000 7f31638667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:19.591 INFO:tasks.workunit.client.0.smithi112.stderr:188683: 2024-09-13T14:38:14.966+0000 7f31638667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:19.591 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:38:19.591 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:38:19.591 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:38:19.591 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:38:19.591 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:38:19.591 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:19.591 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:38:19.591 INFO:tasks.workunit.client.0.smithi112.stderr:188683: 0 2024-09-13T14:38:19.591 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:38:19.592 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:19.592 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:19.592 INFO:tasks.workunit.client.0.smithi112.stderr:188683: 2024-09-13T14:38:16.458+0000 7f31638667c0 -1 Falling back to public interface 2024-09-13T14:38:19.592 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:19.592 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:19.592 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:38:19.592 INFO:tasks.workunit.client.0.smithi112.stderr:188683: 1 2024-09-13T14:38:19.592 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:19.592 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:38:19.592 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:19.592 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:19.592 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:19.592 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:38:19.593 INFO:tasks.workunit.client.0.smithi112.stderr:188683: 2 2024-09-13T14:38:19.593 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:19.593 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:38:19.593 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:19.593 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:19.593 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:19.593 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:38:19.593 INFO:tasks.workunit.client.0.smithi112.stderr:188683: 3 2024-09-13T14:38:19.593 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:19.593 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:38:19.593 INFO:tasks.workunit.client.0.smithi112.stderr:188683: 2024-09-13T14:38:19.534+0000 7f31638667c0 -1 osd.3 85 log_to_monitors true 2024-09-13T14:38:20.303 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers88685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:38:20.303 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:38:20.303 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:38:20.303 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:38:20.303 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:38:20.303 INFO:tasks.workunit.client.0.smithi112.stderr:188685: 2024-09-13T14:38:15.298+0000 7f870f5267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:20.303 INFO:tasks.workunit.client.0.smithi112.stderr:188685: 2024-09-13T14:38:15.310+0000 7f870f5267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:20.304 INFO:tasks.workunit.client.0.smithi112.stderr:188685: 2024-09-13T14:38:15.310+0000 7f870f5267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:20.304 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:38:20.304 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:38:20.304 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:38:20.304 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:38:20.305 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:38:20.305 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:20.305 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:38:20.305 INFO:tasks.workunit.client.0.smithi112.stderr:188685: 0 2024-09-13T14:38:20.305 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:20.305 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:20.306 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:20.306 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:20.306 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:20.306 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:38:20.306 INFO:tasks.workunit.client.0.smithi112.stderr:188685: 1 2024-09-13T14:38:20.306 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:20.306 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:20.307 INFO:tasks.workunit.client.0.smithi112.stderr:188685: 2024-09-13T14:38:17.062+0000 7f870f5267c0 -1 Falling back to public interface 2024-09-13T14:38:20.307 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:20.307 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:20.307 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:20.307 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:38:20.308 INFO:tasks.workunit.client.0.smithi112.stderr:188685: 2 2024-09-13T14:38:20.308 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:20.308 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:20.308 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:20.309 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:20.309 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:20.310 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:38:20.310 INFO:tasks.workunit.client.0.smithi112.stderr:188685: 3 2024-09-13T14:38:20.310 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:20.310 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:20.310 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:22.033 INFO:tasks.workunit.client.0.smithi112.stderr:188685: 2024-09-13T14:38:20.302+0000 7f870f5267c0 -.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:22.033 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:22.033 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:22.033 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:38:22.034 INFO:tasks.workunit.client.0.smithi112.stderr:188683: 4 2024-09-13T14:38:22.034 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:22.034 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:38:22.034 INFO:tasks.workunit.client.0.smithi112.stderr:188683: osd.3 up in weight 1 up_from 90 up_thru 90 down_at 86 last_clean_interval [64,85) [v2:127.0.0.1:6810/3630312,v1:127.0.0.1:6811/3630312] [v2:127.0.0.1:6812/3630312,v1:127.0.0.1:6813/3630312] exists,up 0c26729b-baf1-4c78-9d61-9982d2d70dc2 2024-09-13T14:38:22.034 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:38:22.034 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:38:22.035 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:38:22.035 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:38:22.035 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:38:22.035 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:38:22.035 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:38:22.035 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:38:22.035 INFO:tasks.workunit.client.0.smithi112.stderr:188683: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:38:22.036 INFO:tasks.workunit.client.0.smithi112.stderr:188683: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:38:22.036 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:38:22.036 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:38:22.036 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:38:22.036 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:38:22.037 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:38:22.037 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:38:22.037 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:38:22.038 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:38:22.038 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:38:22.038 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:38:22.038 INFO:tasks.workunit.client.0.smithi112.stderr:188683: 1 2024-09-13T14:38:22.038 INFO:tasks.workunit.client.0.smithi112.stderr:188683: 2 2024-09-13T14:38:22.038 INFO:tasks.workunit.client.0.smithi112.stderr:188683: 3' 2024-09-13T14:38:22.038 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:38:22.038 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:22.038 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:38:22.038 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836547 2024-09-13T14:38:22.039 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836547 2024-09-13T14:38:22.039 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547' 2024-09-13T14:38:22.039 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:22.527 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210:1 osd.1 85 log_to_monitors true 2024-09-13T14:38:22.527 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:22.527 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:22.527 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:38:22.527 INFO:tasks.workunit.client.0.smithi112.stderr:188685: 4 2024-09-13T14:38:22.528 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:22.528 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:22.528 INFO:tasks.workunit.client.0.smithi112.stderr:188685: osd.1 up in weight 1 up_from 92 up_thru 84 down_at 86 last_clean_interval [80,85) [v2:127.0.0.1:6826/1212735600,v1:127.0.0.1:6827/1212735600] [v2:127.0.0.1:6828/1212735600,v1:127.0.0.1:6829/1212735600] exists,up 6bbc9a9f-3432-44d8-a504-eec0cac352ac 2024-09-13T14:38:22.528 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:38:22.528 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:38:22.528 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:38:22.528 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:38:22.528 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:38:22.528 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:38:22.528 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:38:22.528 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:38:22.528 INFO:tasks.workunit.client.0.smithi112.stderr:188685: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:38:22.528 INFO:tasks.workunit.client.0.smithi112.stderr:188685: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:38:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:38:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:38:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:38:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:38:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:38:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:38:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:38:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:38:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:38:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:38:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:188685: 1 2024-09-13T14:38:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:188685: 2 2024-09-13T14:38:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:188685: 3' 2024-09-13T14:38:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:38:22.530 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:22.530 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:38:22.530 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836548 2024-09-13T14:38:22.530 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836548 2024-09-13T14:38:22.530 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548' 2024-09-13T14:38:22.530 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:23.640 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-help flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:38:23.640 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=395136991234 2024-09-13T14:38:23.641 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 395136991234 2024-09-13T14:38:23.641 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-395136991234' 2024-09-13T14:38:23.641 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:23.641 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:38:23.641 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252870 2024-09-13T14:38:23.641 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252870 2024-09-13T14:38:23.641 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-395136991234 2-360777252870' 2024-09-13T14:38:23.641 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:23.641 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:38:23.641 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=386547056642 2024-09-13T14:38:23.641 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 386547056642 2024-09-13T14:38:23.641 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-395136991234 2-360777252870 3-386547056642' 2024-09-13T14:38:23.641 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:23.642 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836547 2024-09-13T14:38:23.642 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:23.642 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:38:23.642 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836547 2024-09-13T14:38:23.642 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:23.642 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836547 2024-09-13T14:38:23.642 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-09-13T14:38:23.642 INFO:tasks.workunit.client.0.smithi112.stderr:188683: waiting osd.0 seq 21474836547 2024-09-13T14:38:23.642 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:38:23.642 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836548 -lt 21474836547 2024-09-13T14:38:23.642 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:23.642 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-395136991234 2024-09-13T14:38:23.642 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:23.642 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:38:23.643 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-395136991234 2024-09-13T14:38:23.643 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:23.643 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=395136991234 2024-09-13T14:38:23.643 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 395136991234' 2024-09-13T14:38:23.643 INFO:tasks.workunit.client.0.smithi112.stderr:188683: waiting osd.1 seq 395136991234 2024-09-13T14:38:23.643 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:38:23.910 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_sers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:38:23.910 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=395136991235 2024-09-13T14:38:23.910 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 395136991235 2024-09-13T14:38:23.911 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-395136991235' 2024-09-13T14:38:23.911 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:23.911 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:38:23.911 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252871 2024-09-13T14:38:23.911 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252871 2024-09-13T14:38:23.911 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-395136991235 2-360777252871' 2024-09-13T14:38:23.911 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:23.911 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:38:23.912 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=386547056643 2024-09-13T14:38:23.912 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 386547056643 2024-09-13T14:38:23.912 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-395136991235 2-360777252871 3-386547056643' 2024-09-13T14:38:23.912 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:23.912 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836548 2024-09-13T14:38:23.912 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:23.912 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:38:23.912 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836548 2024-09-13T14:38:23.912 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:23.912 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836548 2024-09-13T14:38:23.912 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-09-13T14:38:23.912 INFO:tasks.workunit.client.0.smithi112.stderr:188685: waiting osd.0 seq 21474836548 2024-09-13T14:38:23.913 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:38:23.913 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836548 -lt 21474836548 2024-09-13T14:38:23.913 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:23.913 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-395136991235 2024-09-13T14:38:23.913 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:23.913 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:38:23.913 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-395136991235 2024-09-13T14:38:23.913 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:23.913 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=395136991235 2024-09-13T14:38:23.913 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 395136991235' 2024-09-13T14:38:23.913 INFO:tasks.workunit.client.0.smithi112.stderr:188685: waiting osd.1 seq 395136991235 2024-09-13T14:38:23.913 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:38:26.000 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222tats: test 395136991234 -lt 395136991234 2024-09-13T14:38:26.000 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:26.000 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-360777252870 2024-09-13T14:38:26.000 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:26.000 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:38:26.000 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-360777252870 2024-09-13T14:38:26.000 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:26.000 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252870 2024-09-13T14:38:26.000 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 360777252870' 2024-09-13T14:38:26.000 INFO:tasks.workunit.client.0.smithi112.stderr:188683: waiting osd.2 seq 360777252870 2024-09-13T14:38:26.000 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:38:26.001 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252870 -lt 360777252870 2024-09-13T14:38:26.001 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:26.001 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-386547056642 2024-09-13T14:38:26.001 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:26.001 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:38:26.001 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-386547056642 2024-09-13T14:38:26.001 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:26.001 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=386547056642 2024-09-13T14:38:26.001 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 386547056642' 2024-09-13T14:38:26.002 INFO:tasks.workunit.client.0.smithi112.stderr:188683: waiting osd.3 seq 386547056642 2024-09-13T14:38:26.002 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:38:26.002 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 386547056642 2024-09-13T14:38:26.002 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:38:26.002 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:38:26.002 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:38:26.002 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 386547056643 -lt 386547056642 2024-09-13T14:38:26.002 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:38:26.002 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:26.002 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:26.003 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:38:26.003 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:38:26.003 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:38:26.003 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:38:26.003 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:38:26.003 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:38:26.278 INFO:tasks.workunit.client.0.smithi112.stderr:188683: flush_pg_stats: test 395136991234 -lt 395136991235 2024-09-13T14:38:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:38:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:38:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:38:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 395136991235 -lt 395136991235 2024-09-13T14:38:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-360777252871 2024-09-13T14:38:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:38:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-360777252871 2024-09-13T14:38:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252871 2024-09-13T14:38:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 360777252871' 2024-09-13T14:38:26.279 INFO:tasks.workunit.client.0.smithi112.stderr:188685: waiting osd.2 seq 360777252871 2024-09-13T14:38:26.280 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:38:26.280 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252871 -lt 360777252871 2024-09-13T14:38:26.280 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:26.280 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-386547056643 2024-09-13T14:38:26.280 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:26.280 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:38:26.280 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-386547056643 2024-09-13T14:38:26.280 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:26.280 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=386547056643 2024-09-13T14:38:26.280 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 386547056643' 2024-09-13T14:38:26.281 INFO:tasks.workunit.client.0.smithi112.stderr:188685: waiting osd.3 seq 386547056643 2024-09-13T14:38:26.281 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:38:26.281 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 386547056643 -lt 386547056643 2024-09-13T14:38:26.281 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:38:26.281 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:26.281 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:26.281 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:38:26.281 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:38:26.281 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:38:26.281 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:38:26.281 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:38:26.716 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains(: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:38:26.716 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:38:26.716 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:38:26.716 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:38:26.717 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:26.717 INFO:tasks.workunit.client.0.smithi112.stderr:188683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:26.717 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:38:26.717 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:38:26.717 INFO:tasks.workunit.client.0.smithi112.stderr:188683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:38:26.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:38:26.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:38:26.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 188682 2024-09-13T14:38:27.012 INFO:tasks.workunit.client.0.smithi112.stderr:"stale") | not)' 2024-09-13T14:38:27.012 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:38:27.012 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:38:27.012 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:38:27.012 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:38:27.012 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:27.012 INFO:tasks.workunit.client.0.smithi112.stderr:188685: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:27.012 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:38:27.012 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:38:27.012 INFO:tasks.workunit.client.0.smithi112.stderr:188685: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:38:27.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:38:27.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:38:27.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:38:27.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:38:27.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-13T14:38:27.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:38:27.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-13T14:38:27.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:38:27.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:38:27.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:38:27.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:38:27.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-13T14:38:27.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-13T14:38:27.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:38:27.343 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:38:27.343 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:38:27.343 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:38:27.343 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:38:27.343 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:38:27.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:37:53.662789+0000 2024-09-13T14:38:27.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:38:27.946 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:38:27.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:37:53.662789+0000 2024-09-13T14:38:27.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:38:27.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:37:53.662789+0000 2024-09-13T14:38:27.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:38:27.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:38:27.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:38:27.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:38:27.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:38:27.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:38:27.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:38:27.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:38:28.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:37:53.662789+0000 '>' 2024-09-13T14:37:53.662789+0000 2024-09-13T14:38:28.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:38:29.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:38:29.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:38:29.287 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:38:29.287 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:38:29.287 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:38:29.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:38:29.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:38:29.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:37:53.662789+0000 '>' 2024-09-13T14:37:53.662789+0000 2024-09-13T14:38:29.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:38:30.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:38:30.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:38:30.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:38:30.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:38:30.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:38:30.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:38:30.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:38:30.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:38:28.397216+0000 '>' 2024-09-13T14:37:53.662789+0000 2024-09-13T14:38:30.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:38:30.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:38:32.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-13T14:38:32.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:38:32.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:38:32.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:38:32.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 191552"' 2024-09-13T14:38:32.943 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 191552' 2024-09-13T14:38:32.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:38:32.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:38:32.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:38:32.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 191554"' 2024-09-13T14:38:32.943 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 191554' 2024-09-13T14:38:32.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:38:32.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-13T14:38:32.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 191552 191554' 2024-09-13T14:38:32.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:38:32.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/191555: /' 2024-09-13T14:38:32.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:38:32.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 191552 2024-09-13T14:38:32.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/191557: /' 2024-09-13T14:38:32.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:38:34.233 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:38:34.233 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:38:34.233 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:38:34.233 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:38:34.233 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:38:34.233 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:38:34.233 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:38:34.233 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:38:34.234 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:38:34.234 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:38:34.234 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:38:34.234 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:38:34.234 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:38:34.234 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:38:34.234 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:38:34.234 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:38:34.234 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:38:34.234 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:38:34.234 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:38:34.234 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:38:34.234 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:38:34.235 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:38:34.235 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-13T14:38:34.235 INFO:tasks.workunit.client.0.smithi112.stderr:191555: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-13T14:38:34.235 INFO:tasks.workunit.client.0.smithi112.stderr:191555: _ 2024-09-13T14:38:34.235 INFO:tasks.workunit.client.0.smithi112.stderr:191555: hinfo_key 2024-09-13T14:38:34.235 INFO:tasks.workunit.client.0.smithi112.stderr:191555: snapset 2024-09-13T14:38:34.235 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:38:34.235 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:38:34.235 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:38:34.235 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:38:34.235 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:38:34.235 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:38:34.236 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:38:34.237 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:38:34.237 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:38:34.237 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:38:34.237 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:38:34.237 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:38:34.237 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:38:34.237 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:38:34.238 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:38:34.238 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:38:34.238 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:38:34.238 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:38:34.238 INFO:tasks.workunit.client.0.smithi112.stderr:191555: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:38:34.238 INFO:tasks.workunit.client.0.smithi112.stderr:191555: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:38:34.238 INFO:tasks.workunit.client.0.smithi112.stderr:191555: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:38:34.238 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:38:34.238 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:38:34.238 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:38:34.239 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:38:34.239 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:38:34.239 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:38:34.239 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:38:34.239 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:38:34.239 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:38:34.239 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:38:34.239 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:38:34.239 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:38:34.240 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:38:34.240 INFO:tasks.workunit.client.0.smithi112.stderr:191555: start osd.3 2024-09-13T14:38:34.297 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-l191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:38:34.298 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:38:34.298 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:38:34.298 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:38:34.298 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:38:34.298 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:38:34.298 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:38:34.298 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:38:34.298 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:38:34.298 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:38:34.298 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:38:34.298 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:38:34.298 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:38:34.298 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:38:34.299 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:38:34.299 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:38:34.299 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:38:34.299 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:38:34.299 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:38:34.299 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:38:34.299 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:38:34.299 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:38:34.299 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-13T14:38:34.299 INFO:tasks.workunit.client.0.smithi112.stderr:191557: _ 2024-09-13T14:38:34.300 INFO:tasks.workunit.client.0.smithi112.stderr:191557: hinfo_key 2024-09-13T14:38:34.300 INFO:tasks.workunit.client.0.smithi112.stderr:191557: snapset 2024-09-13T14:38:34.300 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T14:38:34.300 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:38:34.300 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:38:34.300 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:38:34.300 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:38:34.300 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:38:34.300 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:38:34.300 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:38:34.301 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:38:34.301 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:38:34.301 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:38:34.301 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:38:34.301 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:38:34.301 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:38:34.301 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:38:34.301 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:38:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:38:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:38:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:191557: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:38:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:191557: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:38:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:191557: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:38:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:38:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:38:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:38:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:38:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:38:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:38:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:38:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:38:34.302 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:38:34.303 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:38:34.303 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:38:34.303 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:38:34.303 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:38:34.303 INFO:tasks.workunit.client.0.smithi112.stderr:191557: start osd.1 2024-09-13T14:38:39.051 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recoen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:38:39.052 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:38:39.052 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:38:39.052 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:38:39.052 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:38:39.052 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:38:39.052 INFO:tasks.workunit.client.0.smithi112.stderr:191555: 2024-09-13T14:38:34.274+0000 7fbda53997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:39.052 INFO:tasks.workunit.client.0.smithi112.stderr:191555: 2024-09-13T14:38:34.302+0000 7fbda53997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:39.052 INFO:tasks.workunit.client.0.smithi112.stderr:191555: 2024-09-13T14:38:34.306+0000 7fbda53997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:39.052 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:38:39.052 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:38:39.052 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:38:39.052 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:38:39.052 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:38:39.053 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:39.053 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:38:39.053 INFO:tasks.workunit.client.0.smithi112.stderr:191555: 0 2024-09-13T14:38:39.053 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:39.053 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:38:39.053 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:39.053 INFO:tasks.workunit.client.0.smithi112.stderr:191555: 2024-09-13T14:38:35.546+0000 7fbda53997c0 -1 Falling back to public interface 2024-09-13T14:38:39.053 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:39.053 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:39.053 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:38:39.053 INFO:tasks.workunit.client.0.smithi112.stderr:191555: 1 2024-09-13T14:38:39.053 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:39.053 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:38:39.054 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:39.054 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:39.054 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:39.054 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:38:39.054 INFO:tasks.workunit.client.0.smithi112.stderr:191555: 2 2024-09-13T14:38:39.054 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:39.054 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:38:39.054 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:39.054 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:39.054 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:39.054 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:38:39.054 INFO:tasks.workunit.client.0.smithi112.stderr:191555: 3 2024-09-13T14:38:39.054 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:39.055 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:38:40.046 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.very_ops 2024-09-13T14:38:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:38:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:38:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:38:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:38:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:38:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:191557: 2024-09-13T14:38:34.330+0000 7f8b8ad647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:191557: 2024-09-13T14:38:34.330+0000 7f8b8ad647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:191557: 2024-09-13T14:38:34.334+0000 7f8b8ad647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:38:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:38:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:38:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:38:40.048 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:38:40.048 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:40.048 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:38:40.048 INFO:tasks.workunit.client.0.smithi112.stderr:191557: 0 2024-09-13T14:38:40.048 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:40.048 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:40.048 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:40.048 INFO:tasks.workunit.client.0.smithi112.stderr:191557: 2024-09-13T14:38:35.682+0000 7f8b8ad647c0 -1 Falling back to public interface 2024-09-13T14:38:40.048 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:40.048 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:40.048 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:38:40.048 INFO:tasks.workunit.client.0.smithi112.stderr:191557: 1 2024-09-13T14:38:40.048 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:38:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:191557: 2 2024-09-13T14:38:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:38:40.049 INFO:tasks.workunit.client.0.smithi112.stderr:191557: 3 2024-09-13T14:38:40.050 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:40.050 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:40.050 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:42.491 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:42.491 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:42.491 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:38:42.491 INFO:tasks.workunit.client.0.smithi112.stderr:191557: 4 2024-09-13T14:38:42.491 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:42.491 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:42.491 INFO:tasks.workunit.client.0.smithi112.stderr:191557: 2024-09-13T14:38:40.157+0000 7f8b8ad647c0 -1 osd.1 94 log_to_monitors true 2024-09-13T14:38:42.491 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:42.491 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:42.491 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:42.491 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:38:42.491 INFO:tasks.workunit.client.0.smithi112.stderr:191557: 5 2024-09-13T14:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:191557: osd.1 up in weight 1 up_from 102 up_thru 92 down_at 95 last_clean_interval [92,94) [v2:127.0.0.1:6826/2562339592,v1:127.0.0.1:6827/2562339592] [v2:127.0.0.1:6828/2562339592,v1:127.0.0.1:6829/2562339592] exists,up 6bbc9a9f-3432-44d8-a504-eec0cac352ac 2024-09-13T14:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:191557: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:191557: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:38:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:38:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:38:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:38:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:38:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:38:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:38:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:38:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:38:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:38:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:191557: 1 2024-09-13T14:38:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:191557: 2 2024-09-13T14:38:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:191557: 3' 2024-09-13T14:38:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:38:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:42.504 INFO:tasks.workunit.client.0.smithi112.stderr:191550/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:191555: 2024-09-13T14:38:39.249+0000 7fbda53997c0 -1 osd.3 94 log_to_monitors true 2024-09-13T14:38:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:38:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:191555: 4 2024-09-13T14:38:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:38:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:38:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:38:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:38:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:191555: 5 2024-09-13T14:38:42.505 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:42.506 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:38:42.506 INFO:tasks.workunit.client.0.smithi112.stderr:191555: osd.3 up in weight 1 up_from 99 up_thru 99 down_at 95 last_clean_interval [90,94) [v2:127.0.0.1:6810/2622243892,v1:127.0.0.1:6811/2622243892] [v2:127.0.0.1:6812/2622243892,v1:127.0.0.1:6813/2622243892] exists,up 0c26729b-baf1-4c78-9d61-9982d2d70dc2 2024-09-13T14:38:42.506 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:38:42.506 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:38:42.506 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:38:42.506 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:38:42.506 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:38:42.506 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:38:42.506 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:38:42.506 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:38:42.506 INFO:tasks.workunit.client.0.smithi112.stderr:191555: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:38:42.506 INFO:tasks.workunit.client.0.smithi112.stderr:191555: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:38:42.506 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:38:42.507 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:38:42.507 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:38:42.507 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:38:42.507 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:38:42.507 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:38:42.507 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:38:42.507 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:38:42.507 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:38:42.507 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:38:42.507 INFO:tasks.workunit.client.0.smithi112.stderr:191555: 1 2024-09-13T14:38:42.508 INFO:tasks.workunit.client.0.smithi112.stderr:191555: 2 2024-09-13T14:38:42.508 INFO:tasks.workunit.client.0.smithi112.stderr:191555: 3' 2024-09-13T14:38:42.508 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:38:45.331 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubunt7: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:38:45.331 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836554 2024-09-13T14:38:45.331 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836554 2024-09-13T14:38:45.331 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554' 2024-09-13T14:38:45.331 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:45.331 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:38:45.331 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=438086664195 2024-09-13T14:38:45.331 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 438086664195 2024-09-13T14:38:45.331 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-438086664195' 2024-09-13T14:38:45.331 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:45.332 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:38:45.332 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252877 2024-09-13T14:38:45.332 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252877 2024-09-13T14:38:45.332 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-438086664195 2-360777252877' 2024-09-13T14:38:45.332 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:45.332 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:38:45.332 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=425201762306 2024-09-13T14:38:45.332 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 425201762306 2024-09-13T14:38:45.332 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-438086664195 2-360777252877 3-425201762306' 2024-09-13T14:38:45.332 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:45.332 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836554 2024-09-13T14:38:45.332 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:45.332 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:38:45.333 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836554 2024-09-13T14:38:45.333 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:45.333 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836554 2024-09-13T14:38:45.333 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2024-09-13T14:38:45.333 INFO:tasks.workunit.client.0.smithi112.stderr:191557: waiting osd.0 seq 21474836554 2024-09-13T14:38:45.333 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:38:45.333 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836552 -lt 21474836554 2024-09-13T14:38:45.333 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:38:45.333 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:38:45.333 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:38:45.333 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836555 -lt 21474836554 2024-09-13T14:38:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.cu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:38:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836553 2024-09-13T14:38:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836553 2024-09-13T14:38:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553' 2024-09-13T14:38:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:38:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=438086664194 2024-09-13T14:38:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 438086664194 2024-09-13T14:38:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-438086664194' 2024-09-13T14:38:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:38:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252876 2024-09-13T14:38:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252876 2024-09-13T14:38:45.335 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-438086664194 2-360777252876' 2024-09-13T14:38:45.335 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:38:45.335 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:38:45.335 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=425201762307 2024-09-13T14:38:45.335 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 425201762307 2024-09-13T14:38:45.335 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-438086664194 2-360777252876 3-425201762307' 2024-09-13T14:38:45.335 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:45.335 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836553 2024-09-13T14:38:45.335 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:45.335 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:38:45.335 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836553 2024-09-13T14:38:45.335 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:45.335 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836553 2024-09-13T14:38:45.336 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-09-13T14:38:45.336 INFO:tasks.workunit.client.0.smithi112.stderr:191555: waiting osd.0 seq 21474836553 2024-09-13T14:38:45.336 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:38:45.336 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836552 -lt 21474836553 2024-09-13T14:38:45.336 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:38:45.336 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:38:45.336 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:38:46.774 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helplient.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:46.774 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-438086664195 2024-09-13T14:38:46.774 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:46.774 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:38:46.774 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-438086664195 2024-09-13T14:38:46.774 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:46.774 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=438086664195 2024-09-13T14:38:46.774 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 438086664195' 2024-09-13T14:38:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:191557: waiting osd.1 seq 438086664195 2024-09-13T14:38:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:38:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 438086664195 -lt 438086664195 2024-09-13T14:38:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-360777252877 2024-09-13T14:38:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:38:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-360777252877 2024-09-13T14:38:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252877 2024-09-13T14:38:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 360777252877' 2024-09-13T14:38:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:191557: waiting osd.2 seq 360777252877 2024-09-13T14:38:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:38:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252877 -lt 360777252877 2024-09-13T14:38:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-425201762306 2024-09-13T14:38:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:38:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-425201762306 2024-09-13T14:38:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=425201762306 2024-09-13T14:38:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 425201762306' 2024-09-13T14:38:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:191557: waiting osd.3 seq 425201762306 2024-09-13T14:38:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:38:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 425201762307 -lt 425201762306 2024-09-13T14:38:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:38:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:46.776 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: testers.sh:2222: flush_pg_stats: test 21474836555 -lt 21474836553 2024-09-13T14:38:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-438086664194 2024-09-13T14:38:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:38:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-438086664194 2024-09-13T14:38:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=438086664194 2024-09-13T14:38:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 438086664194' 2024-09-13T14:38:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:191555: waiting osd.1 seq 438086664194 2024-09-13T14:38:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:38:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 438086664195 -lt 438086664194 2024-09-13T14:38:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-360777252876 2024-09-13T14:38:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:38:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-360777252876 2024-09-13T14:38:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252876 2024-09-13T14:38:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 360777252876' 2024-09-13T14:38:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:191555: waiting osd.2 seq 360777252876 2024-09-13T14:38:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:38:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252877 -lt 360777252876 2024-09-13T14:38:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:38:46.786 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-425201762307 2024-09-13T14:38:46.787 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:38:46.787 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:38:46.787 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-425201762307 2024-09-13T14:38:46.787 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:38:46.787 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=425201762307 2024-09-13T14:38:46.787 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 425201762307' 2024-09-13T14:38:46.787 INFO:tasks.workunit.client.0.smithi112.stderr:191555: waiting osd.3 seq 425201762307 2024-09-13T14:38:46.787 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:38:46.787 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 425201762307 -lt 425201762307 2024-09-13T14:38:46.787 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:38:46.787 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:46.787 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:48.283 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:38:48.284 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:38:48.284 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:38:48.284 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:38:48.284 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:38:48.284 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:38:48.284 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:38:48.284 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:38:48.284 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:38:48.284 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:38:48.284 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:48.285 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:48.285 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-13T14:38:48.285 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-13T14:38:48.285 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T14:38:48.285 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-13T14:38:48.285 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:38:48.285 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:38:48.285 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:38:48.285 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:38:48.285 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:38:48.285 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:38:48.285 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:38:48.285 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:38:48.286 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:38:48.286 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:38:48.286 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:38:48.286 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:38:48.286 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:48.286 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:48.286 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-13T14:38:48.286 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-13T14:38:48.303 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /h 5 == 0 2024-09-13T14:38:48.303 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:38:48.303 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:38:48.303 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:38:48.303 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:38:48.303 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:38:48.303 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:38:48.303 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:38:48.303 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:38:48.303 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:38:48.303 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:48.304 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:48.304 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-13T14:38:48.304 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-13T14:38:48.304 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T14:38:48.304 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-13T14:38:48.304 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:38:48.304 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:38:48.304 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:38:48.304 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:38:48.304 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:38:48.304 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:38:48.304 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:38:48.304 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:38:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:38:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:38:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:38:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:38:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-13T14:38:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-13T14:38:49.460 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_makinome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T14:38:49.460 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T14:38:49.460 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T14:38:49.460 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T14:38:49.460 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T14:38:49.460 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T14:38:49.460 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T14:38:49.460 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=2 2024-09-13T14:38:49.461 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 2 '!=' null 2024-09-13T14:38:49.461 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-13T14:38:49.461 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:38:49.461 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:38:49.461 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:38:49.461 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:38:49.461 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:38:49.461 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:38:49.461 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:38:49.461 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:38:49.461 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:38:49.461 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:38:49.461 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:38:49.461 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:38:49.462 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:49.462 INFO:tasks.workunit.client.0.smithi112.stderr:191555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:49.462 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:38:49.462 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:38:49.462 INFO:tasks.workunit.client.0.smithi112.stderr:191555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:38:49.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:38:49.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:38:49.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 191554 2024-09-13T14:38:49.480 INFO:tasks.workunit.client.0.smithi112.stderr:g_recovery_progress 2024-09-13T14:38:49.480 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T14:38:49.480 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T14:38:49.480 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T14:38:49.480 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T14:38:49.480 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T14:38:49.480 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T14:38:49.480 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=2 2024-09-13T14:38:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 2 '!=' null 2024-09-13T14:38:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-13T14:38:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T14:38:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T14:38:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T14:38:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:38:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:38:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:38:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:38:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:38:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:38:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:38:49.481 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:38:49.482 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:38:49.482 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:38:49.482 INFO:tasks.workunit.client.0.smithi112.stderr:191557: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:38:49.482 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:38:49.482 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:38:49.482 INFO:tasks.workunit.client.0.smithi112.stderr:191557: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:38:49.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:38:49.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:38:49.482 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:38:49.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:38:49.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-13T14:38:49.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:38:49.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:38:49.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:38:49.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T14:38:49.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:38:49.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:38:49.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:38:49.547 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:38:49.547 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:38:49.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:38:49.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:38:49.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:38:49.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:38:49.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:38:49.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:38:49.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:38:49.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:38:49.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:38:49.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:38:49.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:38:49.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:38:49.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:38:49.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:38:49.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:38:49.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:38:49.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:38:49.692 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:38:49.693 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:38:49.693 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:38:49.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:38:49.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:38:49.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:38:49.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T14:38:49.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T14:38:49.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T14:38:49.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T14:38:49.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:38:49.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:38:49.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:38:49.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:38:49.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:38:49.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:38:49.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:38:49.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:38:49.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:38:49.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:38:49.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:38:49.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:38:49.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:38:49.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:38:49.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:38:49.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:38:49.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:38:49.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:38:49.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:38:49.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:38:49.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:38:49.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:38:49.703 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:38:49.703 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:38:49.703 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:38:49.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:38:49.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:38:49.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:38:49.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T14:38:49.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:38:49.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:38:49.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:38:49.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T14:38:49.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:38:49.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T14:38:49.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:38:49.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T14:38:49.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-09-13T14:38:49.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-09-13T14:38:49.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-09-13T14:38:49.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-09-13T14:38:49.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-09-13T14:38:49.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-09-13T14:38:49.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T14:38:49.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T14:38:49.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T14:38:49.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T14:38:49.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T14:38:49.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T14:38:50.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T14:38:50.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:38:50.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:38:50.179 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:38:50.179 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:38:50.179 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:38:50.180 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:38:50.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T14:38:50.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T14:38:50.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T14:38:50.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:38:50.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:38:50.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T14:38:50.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T14:38:50.385 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:38:50.385 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:38:50.385 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:38:50.385 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:38:50.385 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:38:50.385 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:38:50.386 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:38:50.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:38:50.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T14:38:50.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T14:38:50.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:38:50.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:38:50.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T14:38:50.497 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:38:50.497 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T14:38:50.497 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:38:50.497 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:38:50.498 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:38:50.498 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:38:50.498 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:38:50.498 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:38:50.498 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:38:50.498 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T14:38:50.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-09-13T14:38:50.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T14:38:50.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T14:38:50.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T14:38:50.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T14:38:50.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T14:38:50.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T14:38:51.448 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T14:38:51.448 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:38:51.448 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:38:51.449 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:38:51.449 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:38:51.449 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:38:51.449 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:38:51.449 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:38:51.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:38:51.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-09-13T14:38:51.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:38:51.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-09-13T14:38:51.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:38:51.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:38:51.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T14:38:51.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:38:51.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:38:51.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:38:51.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:38:51.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:38:51.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:38:51.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:38:51.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:38:51.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:38:51.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:38:51.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:38:51.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:38:51.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:38:51.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:38:51.493 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:38:51.493 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:38:51.493 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:38:51.494 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:38:51.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:38:51.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:38:51.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:38:51.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:38:51.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:38:51.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:38:51.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:38:51.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:38:51.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:38:51.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:38:51.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:38:51.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:38:51.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:38:51.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:38:51.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=deda4f5c-aa00-43a6-9765-cd3b6256765d 2024-09-13T14:38:51.495 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 deda4f5c-aa00-43a6-9765-cd3b6256765d 2024-09-13T14:38:51.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 deda4f5c-aa00-43a6-9765-cd3b6256765d' 2024-09-13T14:38:51.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:38:51.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB7TuRmzlxrHhAAuJ6QIUA3rr++sIEcp+f+MA== 2024-09-13T14:38:51.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB7TuRmzlxrHhAAuJ6QIUA3rr++sIEcp+f+MA=="}' 2024-09-13T14:38:51.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new deda4f5c-aa00-43a6-9765-cd3b6256765d -i td/osd-scrub-repair/0/new.json 2024-09-13T14:38:51.751 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:38:51.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T14:38:51.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB7TuRmzlxrHhAAuJ6QIUA3rr++sIEcp+f+MA== --osd-uuid deda4f5c-aa00-43a6-9765-cd3b6256765d 2024-09-13T14:38:51.803 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:38:51.801+0000 7fa4e6b6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:51.806 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:38:51.805+0000 7fa4e6b6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:51.807 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:38:51.805+0000 7fa4e6b6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:51.807 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:38:51.805+0000 7fa4e6b6a7c0 -1 bdev(0x5609fa22fc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:38:51.807 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:38:51.805+0000 7fa4e6b6a7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T14:38:58.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T14:38:58.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:38:58.337 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:38:58.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T14:38:58.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:38:58.722 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:38:58.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T14:38:58.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:38:58.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:38:58.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:38:58.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:38:58.753 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:38:58.749+0000 7f0c3a0a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:58.756 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:38:58.753+0000 7f0c3a0a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:58.757 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:38:58.753+0000 7f0c3a0a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:38:59.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T14:38:59.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:38:59.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:38:59.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:38:59.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:38:59.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:38:59.064 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:38:59.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:38:59.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:38:59.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:38:59.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:00.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:00.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:00.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:39:00.429 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:39:00.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:00.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:00.504 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:39:00.501+0000 7f0c3a0a57c0 -1 Falling back to public interface 2024-09-13T14:39:00.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:01.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:01.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:01.763 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:39:01.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:39:01.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:01.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:02.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:03.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:03.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:03.100 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:39:03.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:39:03.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:03.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:03.231 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:39:03.229+0000 7f0c3a0a57c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:39:03.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:04.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:04.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:04.431 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:39:04.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:39:04.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:04.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:04.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:05.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:05.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:05.828 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:39:05.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:39:05.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:05.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:06.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:07.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:07.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:07.261 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:39:07.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:39:07.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:07.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:07.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:08.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:08.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:08.590 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:39:08.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:39:08.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:08.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:09.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:10.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:10.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:10.009 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:39:10.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:39:10.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:10.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:10.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:11.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:11.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:11.340 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:39:11.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:39:11.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:11.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:11.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:12.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:12.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:12.683 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:39:12.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:39:12.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:12.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:13.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:14.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:14.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:14.023 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:39:14.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:39:14.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:14.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:14.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:15.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:15.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:15.349 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:39:15.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:39:15.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:15.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:15.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:16.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:16.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:16.671 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:39:16.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:39:16.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:16.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:17.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:18.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:18.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:18.392 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:39:18.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:39:18.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:18.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:19.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:20.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:20.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:20.303 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:39:20.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:39:20.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:20.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:21.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:22.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:22.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:22.148 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:39:22.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:39:22.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:22.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:22.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:23.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:23.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:23.582 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:39:23.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:39:23.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:23.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:24.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:25.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:25.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:25.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:39:25.338 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:39:25.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:25.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:25.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:26.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:26.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:26.946 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:39:26.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:39:26.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:26.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:39:27.291 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1393506970,v1:127.0.0.1:6803/1393506970] [v2:127.0.0.1:6804/1393506970,v1:127.0.0.1:6805/1393506970] exists,up deda4f5c-aa00-43a6-9765-cd3b6256765d 2024-09-13T14:39:27.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:39:27.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:39:27.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:39:27.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:39:27.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-09-13T14:39:27.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:39:27.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:39:27.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T14:39:27.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:39:27.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:39:27.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:39:27.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:39:27.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:39:27.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:39:27.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:39:27.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:39:27.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:39:27.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:39:27.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:39:27.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:39:27.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:39:27.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:39:27.300 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:39:27.301 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:39:27.301 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:39:27.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:39:27.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:39:27.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:39:27.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:39:27.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:39:27.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:39:27.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:39:27.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:39:27.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:39:27.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:39:27.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:39:27.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:39:27.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:39:27.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:39:27.305 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:39:27.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=35cbf0fd-59af-40ab-8e3b-550420036514 2024-09-13T14:39:27.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 35cbf0fd-59af-40ab-8e3b-550420036514' 2024-09-13T14:39:27.306 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 35cbf0fd-59af-40ab-8e3b-550420036514 2024-09-13T14:39:27.306 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:39:27.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCfTuRmlGY2ExAAYBMcGKTGHfBAcYahz5PZbA== 2024-09-13T14:39:27.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCfTuRmlGY2ExAAYBMcGKTGHfBAcYahz5PZbA=="}' 2024-09-13T14:39:27.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 35cbf0fd-59af-40ab-8e3b-550420036514 -i td/osd-scrub-repair/1/new.json 2024-09-13T14:39:27.722 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:39:27.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T14:39:27.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCfTuRmlGY2ExAAYBMcGKTGHfBAcYahz5PZbA== --osd-uuid 35cbf0fd-59af-40ab-8e3b-550420036514 2024-09-13T14:39:27.776 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:39:27.772+0000 7f48938e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:39:27.798 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:39:27.796+0000 7f48938e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:39:27.800 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:39:27.796+0000 7f48938e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:39:27.800 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:39:27.796+0000 7f48938e87c0 -1 bdev(0x56457d3c3c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:39:27.800 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:39:27.796+0000 7f48938e87c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T14:39:34.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T14:39:34.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:39:34.840 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:39:34.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T14:39:34.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:39:35.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T14:39:35.241 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:39:35.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:39:35.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:39:35.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:39:35.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:39:35.267 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:39:35.268+0000 7fe95a7127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:39:35.270 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:39:35.272+0000 7fe95a7127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:39:35.271 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:39:35.272+0000 7fe95a7127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:39:35.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T14:39:35.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:39:35.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:39:35.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:39:35.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:39:35.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:35.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:39:35.605 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:39:35.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:35.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:35.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:36.549 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:39:36.548+0000 7fe95a7127c0 -1 Falling back to public interface 2024-09-13T14:39:36.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:36.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:36.948 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:39:36.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:39:36.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:36.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:37.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:38.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:38.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:38.271 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:39:38.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:39:38.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:38.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:38.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:39.301 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:39:39.300+0000 7fe95a7127c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:39:39.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:39.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:39.596 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:39:39.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:39:39.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:39.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:40.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:41.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:41.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:41.018 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:39:41.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:39:41.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:41.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:41.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:42.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:42.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:42.357 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:39:42.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:39:42.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:42.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:42.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:43.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:43.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:43.804 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:39:43.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:39:43.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:43.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:44.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:45.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:45.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:45.164 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:39:45.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:39:45.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:45.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:45.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:46.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:46.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:46.501 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:39:46.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:39:46.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:46.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:46.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:47.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:47.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:47.869 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:39:47.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:39:47.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:47.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:48.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:49.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:49.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:49.188 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:39:49.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:39:49.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:49.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:49.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:50.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:50.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:50.517 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:39:50.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:39:50.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:50.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:50.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:51.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:51.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:51.853 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:39:51.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:39:51.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:51.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:52.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:53.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:53.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:53.800 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:39:53.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:39:53.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:53.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:54.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:55.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:55.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:55.854 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:39:55.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:39:55.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:55.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:56.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:57.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:57.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:57.499 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:39:57.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:39:57.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:57.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:39:58.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:39:59.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:39:59.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:39:59.329 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:39:59.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:39:59.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:39:59.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:40:00.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:01.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:01.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:01.032 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:40:01.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:40:01.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:01.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:40:01.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:02.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:02.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:02.435 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:40:02.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:40:02.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:02.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:40:02.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:03.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:03.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:03.894 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:40:03.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:40:03.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:03.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:40:04.231 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4189676319,v1:127.0.0.1:6811/4189676319] [v2:127.0.0.1:6812/4189676319,v1:127.0.0.1:6813/4189676319] exists,up 35cbf0fd-59af-40ab-8e3b-550420036514 2024-09-13T14:40:04.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:40:04.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:40:04.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:40:04.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:40:04.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-09-13T14:40:04.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:40:04.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:40:04.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T14:40:04.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:40:04.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:40:04.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:40:04.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:40:04.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:40:04.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:40:04.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T14:40:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T14:40:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:40:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:40:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:40:04.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:40:04.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:40:04.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:40:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:40:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:40:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:40:04.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:40:04.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:40:04.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:40:04.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:40:04.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:40:04.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:40:04.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:40:04.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:40:04.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:40:04.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:40:04.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:40:04.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:40:04.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:40:04.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T14:40:04.236 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:40:04.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=265dcd70-aa8f-4d1e-a035-54af4ba1410f 2024-09-13T14:40:04.237 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 265dcd70-aa8f-4d1e-a035-54af4ba1410f 2024-09-13T14:40:04.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 265dcd70-aa8f-4d1e-a035-54af4ba1410f' 2024-09-13T14:40:04.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:40:04.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDETuRmR4F7DxAAxLS0CkIHCQsHNjd1eqUXMA== 2024-09-13T14:40:04.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDETuRmR4F7DxAAxLS0CkIHCQsHNjd1eqUXMA=="}' 2024-09-13T14:40:04.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 265dcd70-aa8f-4d1e-a035-54af4ba1410f -i td/osd-scrub-repair/2/new.json 2024-09-13T14:40:04.712 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:40:04.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T14:40:04.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDETuRmR4F7DxAAxLS0CkIHCQsHNjd1eqUXMA== --osd-uuid 265dcd70-aa8f-4d1e-a035-54af4ba1410f 2024-09-13T14:40:04.775 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:04.775+0000 7f59ead4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:40:04.822 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:04.823+0000 7f59ead4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:40:04.824 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:04.823+0000 7f59ead4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:40:04.825 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:04.823+0000 7f59ead4a7c0 -1 bdev(0x55d483177c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T14:40:04.825 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:04.823+0000 7f59ead4a7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T14:40:11.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T14:40:11.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:40:11.005 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T14:40:11.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T14:40:11.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:40:11.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T14:40:11.397 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T14:40:11.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:40:11.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:40:11.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:40:11.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:40:11.429 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:11.427+0000 7f277379d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:40:11.459 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:11.459+0000 7f277379d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:40:11.460 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:11.459+0000 7f277379d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:40:11.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T14:40:11.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:40:11.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:40:11.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:40:11.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:40:11.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:11.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:40:11.737 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:40:11.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:11.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:12.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:12.769 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:12.767+0000 7f277379d7c0 -1 Falling back to public interface 2024-09-13T14:40:13.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:13.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:13.074 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:40:13.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:40:13.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:13.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:13.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:14.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:14.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:14.396 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:40:14.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:40:14.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:14.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:14.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:15.707 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:15.707+0000 7f277379d7c0 -1 osd.2 0 log_to_monitors true 2024-09-13T14:40:15.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:15.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:15.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:40:15.718 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:40:15.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:15.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:16.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:17.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:17.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:17.075 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:40:17.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:40:17.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:17.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:17.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:18.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:18.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:18.466 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:40:18.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:40:18.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:18.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:18.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:19.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:19.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:19.904 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:40:19.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:40:19.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:19.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:20.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:21.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:21.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:21.239 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:40:21.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:40:21.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:21.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:21.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:22.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:22.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:22.625 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:40:22.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:40:22.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:22.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:23.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:24.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:24.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:24.026 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:40:24.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:40:24.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:24.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:24.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:25.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:25.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:25.427 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:40:25.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:40:25.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:25.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:25.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:26.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:26.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:26.850 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:40:26.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:40:26.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:26.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:27.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:28.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:28.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:28.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:40:28.281 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:40:28.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:28.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:28.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:29.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:29.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:29.644 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:40:29.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:40:29.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:29.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:30.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:31.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:31.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:31.149 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:40:31.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:40:31.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:31.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:31.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:32.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:32.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:32.915 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:40:32.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:40:32.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:32.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:33.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:34.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:34.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:34.379 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:40:34.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:40:34.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:34.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:35.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:36.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:36.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:36.203 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:40:36.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:40:36.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:36.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:36.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:37.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:37.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:37.586 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:40:37.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:40:37.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:37.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:38.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:39.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:39.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:39.257 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:40:39.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:40:39.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:39.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:39.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:40.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:40.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:40.987 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T14:40:40.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T14:40:40.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:40.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:40:41.322 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/499057670,v1:127.0.0.1:6819/499057670] [v2:127.0.0.1:6820/499057670,v1:127.0.0.1:6821/499057670] exists,up 265dcd70-aa8f-4d1e-a035-54af4ba1410f 2024-09-13T14:40:41.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:40:41.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:40:41.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:40:41.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:40:41.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-09-13T14:40:41.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:40:41.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:40:41.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-13T14:40:41.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:40:41.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:40:41.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:40:41.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:40:41.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:40:41.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:40:41.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:40:41.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:40:41.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:40:41.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:40:41.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:40:41.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:40:41.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:40:41.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:40:41.327 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:40:41.327 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:40:41.327 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:40:41.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:40:41.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:40:41.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:40:41.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:40:41.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:40:41.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:40:41.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:40:41.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:40:41.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:40:41.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:40:41.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:40:41.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:40:41.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:40:41.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:40:41.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:40:41.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=240beeea-9a32-48d8-9b95-819a8b7fdfbd 2024-09-13T14:40:41.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 240beeea-9a32-48d8-9b95-819a8b7fdfbd' 2024-09-13T14:40:41.329 INFO:tasks.workunit.client.0.smithi112.stdout:add osd3 240beeea-9a32-48d8-9b95-819a8b7fdfbd 2024-09-13T14:40:41.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:40:41.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDpTuRmnsGwFBAATO2VOYIbGdW3X9gqdPuQiw== 2024-09-13T14:40:41.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDpTuRmnsGwFBAATO2VOYIbGdW3X9gqdPuQiw=="}' 2024-09-13T14:40:41.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 240beeea-9a32-48d8-9b95-819a8b7fdfbd -i td/osd-scrub-repair/3/new.json 2024-09-13T14:40:41.800 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:40:41.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-13T14:40:41.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDpTuRmnsGwFBAATO2VOYIbGdW3X9gqdPuQiw== --osd-uuid 240beeea-9a32-48d8-9b95-819a8b7fdfbd 2024-09-13T14:40:41.857 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:41.854+0000 7f1ef07a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:40:41.904 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:41.898+0000 7f1ef07a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:40:41.905 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:41.902+0000 7f1ef07a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:40:41.905 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:41.902+0000 7f1ef07a17c0 -1 bdev(0x561218b9bc00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-13T14:40:41.905 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:41.902+0000 7f1ef07a17c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-13T14:40:48.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-13T14:40:48.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:40:48.945 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd3 key to auth repository 2024-09-13T14:40:48.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-13T14:40:48.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:40:49.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-13T14:40:49.363 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.3 2024-09-13T14:40:49.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:40:49.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:40:49.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:40:49.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:40:49.396 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:49.394+0000 7f1121f8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:40:49.432 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:49.430+0000 7f1121f8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:40:49.433 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:49.430+0000 7f1121f8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:40:49.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-13T14:40:49.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:40:49.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:40:49.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:40:49.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:40:49.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:49.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:40:49.702 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:40:49.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:49.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:40:50.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:50.477 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:50.474+0000 7f1121f8c7c0 -1 Falling back to public interface 2024-09-13T14:40:51.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:51.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:51.029 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:40:51.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:40:51.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:51.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:40:51.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:52.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:52.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:52.366 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:40:52.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:40:52.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:52.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:40:52.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:53.369 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:40:53.366+0000 7f1121f8c7c0 -1 osd.3 0 log_to_monitors true 2024-09-13T14:40:53.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:53.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:53.703 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:40:53.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:40:53.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:53.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:40:54.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:55.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:55.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:55.092 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:40:55.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:40:55.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:55.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:40:55.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:56.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:56.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:56.474 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:40:56.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:40:56.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:56.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:40:56.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:57.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:57.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:57.857 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:40:57.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:40:57.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:57.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:40:58.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:40:59.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:40:59.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:40:59.193 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:40:59.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:40:59.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:40:59.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:40:59.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:00.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:00.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:00.590 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:41:00.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:41:00.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:00.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:41:01.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:02.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:02.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:02.059 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:41:02.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:41:02.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:02.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:41:02.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:03.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:03.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:03.417 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:41:03.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:41:03.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:03.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:41:03.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:04.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:04.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:04.750 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:41:04.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:41:04.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:04.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:41:05.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:06.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:06.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:06.078 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:41:06.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:41:06.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:06.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:41:06.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:07.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:07.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:07.460 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:41:07.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:41:07.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:07.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:41:08.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:09.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:09.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:09.092 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:41:09.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:41:09.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:09.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:41:09.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:10.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:10.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:10.894 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:41:10.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:41:10.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:10.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:41:11.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:12.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:12.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:12.561 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:41:12.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:41:12.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:12.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:41:13.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:14.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:14.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:14.362 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:41:14.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:41:14.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:14.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:41:15.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:16.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:16.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:16.148 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:41:16.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:41:16.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:16.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:41:17.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:18.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:18.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:18.016 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:41:18.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:41:18.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:18.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:41:18.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:19.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:19.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:19.404 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T14:41:19.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T14:41:19.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:19.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:41:19.749 INFO:tasks.workunit.client.0.smithi112.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/996906984,v1:127.0.0.1:6827/996906984] [v2:127.0.0.1:6828/996906984,v1:127.0.0.1:6829/996906984] exists,up 240beeea-9a32-48d8-9b95-819a8b7fdfbd 2024-09-13T14:41:19.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:41:19.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:41:19.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:41:19.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:41:19.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-09-13T14:41:19.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:41:19.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:41:19.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-13T14:41:19.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:41:19.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/4 2024-09-13T14:41:19.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:41:19.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:41:19.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:41:19.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:41:19.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-09-13T14:41:19.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-09-13T14:41:19.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:41:19.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:41:19.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:41:19.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:41:19.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:41:19.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:41:19.755 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:41:19.755 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:41:19.755 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:41:19.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:41:19.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:41:19.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:41:19.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:41:19.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:41:19.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:41:19.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:41:19.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:41:19.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:41:19.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:41:19.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:41:19.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:41:19.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:41:19.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/4 2024-09-13T14:41:19.757 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:41:19.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6a8f99b3-5ccf-4271-9103-8366d237bfbe 2024-09-13T14:41:19.757 INFO:tasks.workunit.client.0.smithi112.stdout:add osd4 6a8f99b3-5ccf-4271-9103-8366d237bfbe 2024-09-13T14:41:19.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 6a8f99b3-5ccf-4271-9103-8366d237bfbe' 2024-09-13T14:41:19.758 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:41:19.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAPT+RmJNgfLhAA0TvD7eoKKg32K8M4/wNmgQ== 2024-09-13T14:41:19.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAPT+RmJNgfLhAA0TvD7eoKKg32K8M4/wNmgQ=="}' 2024-09-13T14:41:19.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6a8f99b3-5ccf-4271-9103-8366d237bfbe -i td/osd-scrub-repair/4/new.json 2024-09-13T14:41:20.151 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:41:20.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/4/new.json 2024-09-13T14:41:20.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPT+RmJNgfLhAA0TvD7eoKKg32K8M4/wNmgQ== --osd-uuid 6a8f99b3-5ccf-4271-9103-8366d237bfbe 2024-09-13T14:41:20.207 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:41:20.205+0000 7f4b1f2047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:41:20.236 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:41:20.233+0000 7f4b1f2047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:41:20.237 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:41:20.233+0000 7f4b1f2047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:41:20.238 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:41:20.237+0000 7f4b1f2047c0 -1 bdev(0x563526852800 td/osd-scrub-repair/4/block) open stat got: (1) Operation not permitted 2024-09-13T14:41:20.238 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:41:20.237+0000 7f4b1f2047c0 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-09-13T14:41:26.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-09-13T14:41:26.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:41:26.927 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd4 key to auth repository 2024-09-13T14:41:26.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-13T14:41:26.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:41:27.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-13T14:41:27.340 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.4 2024-09-13T14:41:27.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:41:27.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:41:27.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:41:27.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:41:27.373 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:41:27.369+0000 7f7cf2e257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:41:27.403 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:41:27.401+0000 7f7cf2e257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:41:27.404 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:41:27.401+0000 7f7cf2e257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:41:27.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-13T14:41:27.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:41:27.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-13T14:41:27.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:41:27.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:41:27.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:27.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:41:27.695 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:41:27.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:27.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:28.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:28.677 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:41:28.673+0000 7f7cf2e257c0 -1 Falling back to public interface 2024-09-13T14:41:29.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:29.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:29.041 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:41:29.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:41:29.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:29.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:29.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:30.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:30.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:30.362 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:41:30.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:41:30.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:30.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:30.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:31.469 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:41:31.465+0000 7f7cf2e257c0 -1 osd.4 0 log_to_monitors true 2024-09-13T14:41:31.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:31.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:31.702 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:41:31.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:41:31.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:31.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:32.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:33.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:33.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:33.156 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:41:33.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:41:33.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:33.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:33.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:34.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:34.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:34.502 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:41:34.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:41:34.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:34.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:35.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:35.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:35.880 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:41:35.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:41:35.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:35.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:36.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:37.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:37.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:37.320 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:41:37.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:41:37.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:37.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:37.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:38.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:38.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:38.659 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:41:38.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:41:38.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:38.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:38.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:39.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:39.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:39.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:41:39.991 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:41:39.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:39.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:40.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:41.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:41.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:41.393 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:41:41.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:41:41.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:41.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:41.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:42.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:42.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:42.737 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:41:42.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:41:42.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:42.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:43.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:44.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:44.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:44.076 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:41:44.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:41:44.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:44.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:44.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:45.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:45.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:45.492 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:41:45.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:41:45.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:45.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:45.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:46.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:46.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:46.828 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:41:46.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:41:46.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:46.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:47.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:48.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:48.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:48.232 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:41:48.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:41:48.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:48.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:48.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:49.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:49.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:49.652 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:41:49.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:41:49.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:49.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:50.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:51.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:51.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:51.408 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:41:51.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:41:51.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:51.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:52.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:53.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:53.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:53.093 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:41:53.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:41:53.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:53.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:53.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:54.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:54.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:54.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:41:54.644 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:41:54.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:54.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:55.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:41:56.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:41:56.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:41:56.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T14:41:56.556 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T14:41:56.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:41:56.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:41:57.013 INFO:tasks.workunit.client.0.smithi112.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3848550692,v1:127.0.0.1:6835/3848550692] [v2:127.0.0.1:6836/3848550692,v1:127.0.0.1:6837/3848550692] exists,up 6a8f99b3-5ccf-4271-9103-8366d237bfbe 2024-09-13T14:41:57.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:41:57.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:41:57.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:41:57.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:41:57.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-09-13T14:41:57.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:41:57.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:41:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-13T14:41:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:41:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:41:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:41:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:41:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:41:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:41:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-13T14:41:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-13T14:41:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:41:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:41:57.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:41:57.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:41:57.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:41:57.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:41:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:41:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:41:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:41:57.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:41:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:41:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:41:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:41:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:41:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:41:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:41:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:41:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:41:57.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:41:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:41:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:41:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:41:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/5 2024-09-13T14:41:57.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:41:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ac5b44cf-f8cb-4d05-8d84-3858f28ea531 2024-09-13T14:41:57.021 INFO:tasks.workunit.client.0.smithi112.stdout:add osd5 ac5b44cf-f8cb-4d05-8d84-3858f28ea531 2024-09-13T14:41:57.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 ac5b44cf-f8cb-4d05-8d84-3858f28ea531' 2024-09-13T14:41:57.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:41:57.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA1T+RmRLJlAhAAeVsBC6xPiBdpMZud5EYi2w== 2024-09-13T14:41:57.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA1T+RmRLJlAhAAeVsBC6xPiBdpMZud5EYi2w=="}' 2024-09-13T14:41:57.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ac5b44cf-f8cb-4d05-8d84-3858f28ea531 -i td/osd-scrub-repair/5/new.json 2024-09-13T14:41:57.417 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:41:57.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/5/new.json 2024-09-13T14:41:57.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1T+RmRLJlAhAAeVsBC6xPiBdpMZud5EYi2w== --osd-uuid ac5b44cf-f8cb-4d05-8d84-3858f28ea531 2024-09-13T14:41:57.468 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:41:57.468+0000 7f1ba9c857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:41:57.519 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:41:57.520+0000 7f1ba9c857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:41:57.520 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:41:57.520+0000 7f1ba9c857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:41:57.521 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:41:57.520+0000 7f1ba9c857c0 -1 bdev(0x5564262c1c00 td/osd-scrub-repair/5/block) open stat got: (1) Operation not permitted 2024-09-13T14:41:57.521 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:41:57.520+0000 7f1ba9c857c0 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-09-13T14:42:03.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-09-13T14:42:03.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:42:03.969 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd5 key to auth repository 2024-09-13T14:42:03.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-13T14:42:03.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:42:04.364 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.5 2024-09-13T14:42:04.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-13T14:42:04.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:42:04.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:42:04.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:42:04.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:42:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:42:04.392+0000 7f27682567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:42:04.393 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:42:04.392+0000 7f27682567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:42:04.394 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:42:04.396+0000 7f27682567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:42:04.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-13T14:42:04.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:42:04.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-13T14:42:04.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:42:04.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:42:04.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:04.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:42:04.710 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:42:04.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:04.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:05.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:05.485 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:42:05.484+0000 7f27682567c0 -1 Falling back to public interface 2024-09-13T14:42:06.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:06.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:06.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:42:06.058 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:42:06.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:06.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:06.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:07.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:07.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:07.396 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:42:07.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:42:07.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:07.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:07.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:08.185 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:42:08.184+0000 7f27682567c0 -1 osd.5 0 log_to_monitors true 2024-09-13T14:42:08.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:08.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:08.723 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:42:08.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:42:08.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:08.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:09.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:10.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:10.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:10.317 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:42:10.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:42:10.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:10.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:10.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:11.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:11.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:11.674 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:42:11.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:42:11.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:11.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:11.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:12.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:12.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:12.997 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:42:12.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:42:12.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:12.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:13.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:14.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:14.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:14.479 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:42:14.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:42:14.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:14.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:14.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:15.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:15.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:15.894 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:42:15.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:42:15.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:15.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:16.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:17.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:17.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:17.337 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:42:17.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:42:17.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:17.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:17.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:18.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:18.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:18.673 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:42:18.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:42:18.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:18.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:19.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:20.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:20.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:20.111 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:42:20.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:42:20.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:20.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:20.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:21.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:21.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:21.488 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:42:21.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:42:21.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:21.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:21.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:22.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:22.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:22.819 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:42:22.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:42:22.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:22.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:23.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:24.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:24.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:24.176 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:42:24.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:42:24.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:24.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:25.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:26.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:26.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:26.193 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:42:26.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:42:26.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:26.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:26.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:27.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:27.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:27.712 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:42:27.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:42:27.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:27.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:28.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:29.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:29.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:29.224 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:42:29.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:42:29.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:29.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:30.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:31.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:31.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:31.058 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:42:31.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:42:31.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:31.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:31.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:32.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:32.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:32.688 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:42:32.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:42:32.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:32.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:42:33.091 INFO:tasks.workunit.client.0.smithi112.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3367987343,v1:127.0.0.1:6843/3367987343] [v2:127.0.0.1:6844/3367987343,v1:127.0.0.1:6845/3367987343] exists,up ac5b44cf-f8cb-4d05-8d84-3858f28ea531 2024-09-13T14:42:33.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:42:33.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:42:33.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:42:33.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:42:33.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-09-13T14:42:33.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:42:33.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:42:33.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-13T14:42:33.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:42:33.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/6 2024-09-13T14:42:33.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:42:33.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:42:33.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:42:33.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:42:33.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-09-13T14:42:33.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-09-13T14:42:33.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:42:33.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:42:33.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:42:33.095 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:42:33.095 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:42:33.095 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:42:33.095 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:42:33.095 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:42:33.095 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:42:33.095 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:42:33.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:42:33.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:42:33.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:42:33.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:42:33.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:42:33.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:42:33.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:42:33.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:42:33.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:42:33.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:42:33.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:42:33.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:42:33.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/6 2024-09-13T14:42:33.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:42:33.097 INFO:tasks.workunit.client.0.smithi112.stdout:add osd6 18b1b698-7a21-4d19-b01d-4240c0b9fa12 2024-09-13T14:42:33.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=18b1b698-7a21-4d19-b01d-4240c0b9fa12 2024-09-13T14:42:33.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 18b1b698-7a21-4d19-b01d-4240c0b9fa12' 2024-09-13T14:42:33.098 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:42:33.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBZT+RmKMAaBxAAx80ZLe+k8tx8gwrzXS1zYw== 2024-09-13T14:42:33.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBZT+RmKMAaBxAAx80ZLe+k8tx8gwrzXS1zYw=="}' 2024-09-13T14:42:33.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 18b1b698-7a21-4d19-b01d-4240c0b9fa12 -i td/osd-scrub-repair/6/new.json 2024-09-13T14:42:33.532 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:42:33.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/6/new.json 2024-09-13T14:42:33.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZT+RmKMAaBxAAx80ZLe+k8tx8gwrzXS1zYw== --osd-uuid 18b1b698-7a21-4d19-b01d-4240c0b9fa12 2024-09-13T14:42:33.589 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:42:33.587+0000 7fb9acac97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:42:33.625 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:42:33.623+0000 7fb9acac97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:42:33.627 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:42:33.627+0000 7fb9acac97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:42:33.627 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:42:33.627+0000 7fb9acac97c0 -1 bdev(0x55da9f69e800 td/osd-scrub-repair/6/block) open stat got: (1) Operation not permitted 2024-09-13T14:42:33.627 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:42:33.627+0000 7fb9acac97c0 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-09-13T14:42:39.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-09-13T14:42:39.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:42:39.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-13T14:42:39.958 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd6 key to auth repository 2024-09-13T14:42:39.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:42:40.365 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.6 2024-09-13T14:42:40.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-13T14:42:40.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:42:40.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:42:40.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:42:40.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:42:40.400 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:42:40.399+0000 7fa5b0bce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:42:40.427 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:42:40.427+0000 7fa5b0bce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:42:40.429 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:42:40.427+0000 7fa5b0bce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:42:40.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-13T14:42:40.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:42:40.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-13T14:42:40.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:42:40.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:42:40.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:40.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:42:40.712 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:42:40.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:40.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:42:41.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:41.709 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:42:41.707+0000 7fa5b0bce7c0 -1 Falling back to public interface 2024-09-13T14:42:42.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:42.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:42.033 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:42:42.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:42:42.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:42.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:42:42.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:43.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:43.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:43.357 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:42:43.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:42:43.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:43.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:42:43.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:44.469 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:42:44.467+0000 7fa5b0bce7c0 -1 osd.6 0 log_to_monitors true 2024-09-13T14:42:44.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:44.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:44.691 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:42:44.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:42:44.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:44.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:42:45.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:46.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:46.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:46.210 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:42:46.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:42:46.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:46.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:42:46.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:47.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:47.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:47.620 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:42:47.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:42:47.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:47.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:42:47.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:48.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:48.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:48.961 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:42:48.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:42:48.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:48.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:42:49.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:50.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:50.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:50.346 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:42:50.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:42:50.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:50.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:42:50.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:51.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:51.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:51.690 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:42:51.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:42:51.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:51.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:42:52.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:53.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:53.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:53.029 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:42:53.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:42:53.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:53.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:42:53.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:54.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:54.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:54.371 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:42:54.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:42:54.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:54.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:42:54.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:55.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:55.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:55.853 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:42:55.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:42:55.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:55.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:42:56.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:57.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:57.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:57.196 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:42:57.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:42:57.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:57.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:42:57.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:42:58.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:42:58.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:42:58.764 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:42:58.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:42:58.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:42:58.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:42:59.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:00.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:00.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:00.718 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:43:00.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:43:00.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:00.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:43:01.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:02.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:02.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:02.495 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:43:02.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:43:02.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:02.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:43:03.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:04.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:04.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:04.342 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:43:04.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:43:04.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:04.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:43:04.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:05.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:05.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:05.748 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:43:05.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:43:05.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:05.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:43:06.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:07.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:07.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:07.341 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:43:07.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:43:07.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:07.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:43:07.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:08.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:08.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:08.772 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:43:08.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:43:08.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:08.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:43:09.109 INFO:tasks.workunit.client.0.smithi112.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1986672235,v1:127.0.0.1:6851/1986672235] [v2:127.0.0.1:6852/1986672235,v1:127.0.0.1:6853/1986672235] exists,up 18b1b698-7a21-4d19-b01d-4240c0b9fa12 2024-09-13T14:43:09.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:43:09.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:43:09.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:43:09.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:43:09.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-09-13T14:43:09.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:43:09.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:43:09.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-13T14:43:09.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:43:09.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/7 2024-09-13T14:43:09.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:43:09.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:43:09.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:43:09.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:43:09.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-09-13T14:43:09.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-09-13T14:43:09.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:43:09.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:43:09.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:43:09.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:43:09.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:43:09.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:43:09.114 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:43:09.114 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:43:09.114 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:43:09.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:43:09.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:43:09.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:43:09.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:43:09.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:43:09.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:43:09.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:43:09.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:43:09.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:43:09.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:43:09.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:43:09.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:43:09.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:43:09.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/7 2024-09-13T14:43:09.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:43:09.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=305227fa-6d91-4299-84d7-ed5f4adbb7e0 2024-09-13T14:43:09.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 305227fa-6d91-4299-84d7-ed5f4adbb7e0' 2024-09-13T14:43:09.116 INFO:tasks.workunit.client.0.smithi112.stdout:add osd7 305227fa-6d91-4299-84d7-ed5f4adbb7e0 2024-09-13T14:43:09.117 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:43:09.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB9T+RmIcYvCBAAuLHSA3jZE0AaXh2QxberQw== 2024-09-13T14:43:09.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB9T+RmIcYvCBAAuLHSA3jZE0AaXh2QxberQw=="}' 2024-09-13T14:43:09.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 305227fa-6d91-4299-84d7-ed5f4adbb7e0 -i td/osd-scrub-repair/7/new.json 2024-09-13T14:43:09.486 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:43:09.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/7/new.json 2024-09-13T14:43:09.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9T+RmIcYvCBAAuLHSA3jZE0AaXh2QxberQw== --osd-uuid 305227fa-6d91-4299-84d7-ed5f4adbb7e0 2024-09-13T14:43:09.540 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:09.538+0000 7fb3686dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:43:09.579 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:09.578+0000 7fb3686dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:43:09.581 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:09.578+0000 7fb3686dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:43:09.581 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:09.578+0000 7fb3686dd7c0 -1 bdev(0x55f23513bc00 td/osd-scrub-repair/7/block) open stat got: (1) Operation not permitted 2024-09-13T14:43:09.581 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:09.578+0000 7fb3686dd7c0 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-09-13T14:43:15.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-09-13T14:43:15.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:43:15.973 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd7 key to auth repository 2024-09-13T14:43:15.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-13T14:43:15.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:43:16.386 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.7 2024-09-13T14:43:16.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-13T14:43:16.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:43:16.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:43:16.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:43:16.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:43:16.419 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:16.418+0000 7fd4a6f8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:43:16.440 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:16.438+0000 7fd4a6f8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:43:16.441 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:16.438+0000 7fd4a6f8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:43:16.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-13T14:43:16.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:43:16.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-13T14:43:16.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:43:16.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:43:16.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:16.703 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:43:16.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:43:16.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:16.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:17.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:17.737 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:17.734+0000 7fd4a6f8f7c0 -1 Falling back to public interface 2024-09-13T14:43:18.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:18.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:18.053 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:43:18.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:43:18.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:18.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:18.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:19.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:19.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:19.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:43:19.384 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:43:19.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:19.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:19.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:20.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:20.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:20.720 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:43:20.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:43:20.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:20.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:20.784 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:20.782+0000 7fd4a6f8f7c0 -1 osd.7 0 log_to_monitors true 2024-09-13T14:43:21.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:22.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:22.102 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:43:22.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:22.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:43:22.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:22.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:22.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:23.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:23.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:23.520 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:43:23.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:43:23.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:23.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:23.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:24.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:24.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:24.862 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:43:24.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:43:24.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:24.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:25.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:26.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:26.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:26.248 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:43:26.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:43:26.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:26.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:26.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:27.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:27.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:27.633 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:43:27.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:43:27.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:27.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:27.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:28.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:28.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:28.969 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:43:28.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:43:28.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:28.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:29.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:30.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:30.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:30.309 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:43:30.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:43:30.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:30.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:30.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:31.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:31.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:31.677 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:43:31.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:43:31.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:31.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:31.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:32.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:32.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:32.999 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:43:32.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:43:32.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:32.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:33.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:34.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:34.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:34.318 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:43:34.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:43:34.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:34.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:35.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:36.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:36.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:36.088 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:43:36.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:43:36.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:36.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:36.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:37.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:37.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:37.841 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:43:37.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:43:37.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:37.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:38.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:39.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:39.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:39.522 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:43:39.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:43:39.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:39.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:40.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:41.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:41.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:41.293 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:43:41.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:43:41.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:41.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:41.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:42.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:42.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:42.712 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:43:42.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:43:42.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:42.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:43.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:44.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:44.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:44.528 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:43:44.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:43:44.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:44.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:43:44.965 INFO:tasks.workunit.client.0.smithi112.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2495523649,v1:127.0.0.1:6859/2495523649] [v2:127.0.0.1:6860/2495523649,v1:127.0.0.1:6861/2495523649] exists,up 305227fa-6d91-4299-84d7-ed5f4adbb7e0 2024-09-13T14:43:44.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:43:44.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:43:44.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:43:44.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:43:44.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-09-13T14:43:44.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:43:44.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:43:44.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2024-09-13T14:43:44.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:43:44.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/8 2024-09-13T14:43:44.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:43:44.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:43:44.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:43:44.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:43:44.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-09-13T14:43:44.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-09-13T14:43:44.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:43:44.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:43:44.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:43:44.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:43:44.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:43:44.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:43:44.971 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:43:44.971 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:43:44.971 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:43:44.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:43:44.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:43:44.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:43:44.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:43:44.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:43:44.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:43:44.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:43:44.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:43:44.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:43:44.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:43:44.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:43:44.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:43:44.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:43:44.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/8 2024-09-13T14:43:44.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:43:44.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=23983a08-8d29-484f-8a52-3631cdc0c07d 2024-09-13T14:43:44.973 INFO:tasks.workunit.client.0.smithi112.stdout:add osd8 23983a08-8d29-484f-8a52-3631cdc0c07d 2024-09-13T14:43:44.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 23983a08-8d29-484f-8a52-3631cdc0c07d' 2024-09-13T14:43:44.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:43:44.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCgT+Rmkm9mOxAAXYx8oJfYqnHI/ZuBvqpGgA== 2024-09-13T14:43:44.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCgT+Rmkm9mOxAAXYx8oJfYqnHI/ZuBvqpGgA=="}' 2024-09-13T14:43:44.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 23983a08-8d29-484f-8a52-3631cdc0c07d -i td/osd-scrub-repair/8/new.json 2024-09-13T14:43:45.361 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:43:45.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/8/new.json 2024-09-13T14:43:45.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgT+Rmkm9mOxAAXYx8oJfYqnHI/ZuBvqpGgA== --osd-uuid 23983a08-8d29-484f-8a52-3631cdc0c07d 2024-09-13T14:43:45.409 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:45.405+0000 7fcab25257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:43:45.471 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:45.469+0000 7fcab25257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:43:45.473 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:45.469+0000 7fcab25257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:43:45.473 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:45.469+0000 7fcab25257c0 -1 bdev(0x55818c917c00 td/osd-scrub-repair/8/block) open stat got: (1) Operation not permitted 2024-09-13T14:43:45.473 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:45.469+0000 7fcab25257c0 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-09-13T14:43:52.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-09-13T14:43:52.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:43:52.393 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd8 key to auth repository 2024-09-13T14:43:52.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2024-09-13T14:43:52.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:43:52.805 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.8 2024-09-13T14:43:52.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2024-09-13T14:43:52.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:43:52.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:43:52.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:43:52.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:43:52.837 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:52.833+0000 7f4c4ac137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:43:52.862 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:52.857+0000 7f4c4ac137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:43:52.863 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:52.861+0000 7f4c4ac137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:43:53.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2024-09-13T14:43:53.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:43:53.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2024-09-13T14:43:53.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:43:53.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:43:53.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:53.142 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:43:53.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:43:53.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:53.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:43:53.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:54.201 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:54.197+0000 7f4c4ac137c0 -1 Falling back to public interface 2024-09-13T14:43:54.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:54.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:54.478 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:43:54.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:43:54.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:54.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:43:54.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:55.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:55.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:55.848 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:43:55.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:43:55.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:55.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:43:56.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:57.004 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:43:57.001+0000 7f4c4ac137c0 -1 osd.8 0 log_to_monitors true 2024-09-13T14:43:57.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:57.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:57.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:43:57.197 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:43:57.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:57.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:43:57.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:43:58.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:43:58.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:43:58.539 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:43:58.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:43:58.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:43:58.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:43:59.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:00.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:00.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:00.009 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:44:00.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:44:00.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:00.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:44:00.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:01.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:01.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:01.350 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:44:01.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:44:01.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:01.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:44:01.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:02.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:02.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:02.678 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:44:02.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:44:02.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:02.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:44:03.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:04.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:04.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:04.085 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:44:04.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:44:04.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:04.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:44:04.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:05.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:05.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:05.404 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:44:05.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:44:05.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:05.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:44:05.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:06.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:06.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:06.805 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:44:06.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:44:06.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:06.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:44:07.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:08.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:08.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:08.143 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:44:08.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:44:08.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:08.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:44:08.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:09.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:09.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:09.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:44:09.523 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:44:09.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:09.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:44:10.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:11.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:11.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:11.339 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:44:11.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:44:11.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:11.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:44:12.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:13.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:13.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:13.232 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:44:13.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:44:13.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:13.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:44:14.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:15.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:15.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:15.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:44:15.043 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:44:15.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:15.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:44:15.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:16.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:16.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:16.851 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:44:16.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:44:16.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:16.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:44:17.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:18.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:18.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:18.468 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:44:18.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:44:18.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:18.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:44:19.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:20.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:20.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:20.166 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:44:20.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:44:20.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:20.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:44:20.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:21.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:21.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:21.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:44:21.522 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:44:21.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:21.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:44:21.855 INFO:tasks.workunit.client.0.smithi112.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/1871466184,v1:127.0.0.1:6867/1871466184] [v2:127.0.0.1:6868/1871466184,v1:127.0.0.1:6869/1871466184] exists,up 23983a08-8d29-484f-8a52-3631cdc0c07d 2024-09-13T14:44:21.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:44:21.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:44:21.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:44:21.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:44:21.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-09-13T14:44:21.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:44:21.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:44:21.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2024-09-13T14:44:21.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:44:21.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:44:21.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:44:21.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:44:21.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:44:21.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:44:21.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-13T14:44:21.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-13T14:44:21.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:44:21.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:44:21.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:44:21.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:44:21.860 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:44:21.860 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:44:21.860 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:44:21.860 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:44:21.860 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:44:21.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:44:21.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:44:21.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:44:21.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:44:21.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:44:21.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:44:21.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:44:21.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:44:21.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:44:21.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:44:21.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:44:21.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:44:21.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:44:21.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/9 2024-09-13T14:44:21.862 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:44:21.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=773da5cd-70ef-4087-bab9-35ba28113c89 2024-09-13T14:44:21.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 773da5cd-70ef-4087-bab9-35ba28113c89' 2024-09-13T14:44:21.863 INFO:tasks.workunit.client.0.smithi112.stdout:add osd9 773da5cd-70ef-4087-bab9-35ba28113c89 2024-09-13T14:44:21.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:44:21.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDFT+Rm2JW/NBAAyWBFIEyfyLMNyfFDlpHClw== 2024-09-13T14:44:21.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDFT+Rm2JW/NBAAyWBFIEyfyLMNyfFDlpHClw=="}' 2024-09-13T14:44:21.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 773da5cd-70ef-4087-bab9-35ba28113c89 -i td/osd-scrub-repair/9/new.json 2024-09-13T14:44:22.269 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:44:22.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/9/new.json 2024-09-13T14:44:22.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDFT+Rm2JW/NBAAyWBFIEyfyLMNyfFDlpHClw== --osd-uuid 773da5cd-70ef-4087-bab9-35ba28113c89 2024-09-13T14:44:22.326 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:44:22.324+0000 7fa093f317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:44:22.363 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:44:22.364+0000 7fa093f317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:44:22.364 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:44:22.364+0000 7fa093f317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:44:22.365 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:44:22.364+0000 7fa093f317c0 -1 bdev(0x559381d4fc00 td/osd-scrub-repair/9/block) open stat got: (1) Operation not permitted 2024-09-13T14:44:22.365 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:44:22.364+0000 7fa093f317c0 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-09-13T14:44:29.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-09-13T14:44:29.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:44:29.109 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd9 key to auth repository 2024-09-13T14:44:29.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2024-09-13T14:44:29.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:44:29.528 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.9 2024-09-13T14:44:29.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2024-09-13T14:44:29.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:44:29.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:44:29.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:44:29.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:44:29.562 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:44:29.560+0000 7f96984497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:44:29.592 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:44:29.592+0000 7f96984497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:44:29.593 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:44:29.592+0000 7f96984497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:44:29.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2024-09-13T14:44:29.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:44:29.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-13T14:44:29.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:44:29.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:44:29.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:29.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:44:29.876 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:44:29.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:29.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:30.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:30.861 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:44:30.860+0000 7f96984497c0 -1 Falling back to public interface 2024-09-13T14:44:31.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:31.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:31.208 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:44:31.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:44:31.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:31.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:31.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:32.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:32.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:32.550 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:44:32.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:44:32.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:32.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:32.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:33.827 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:44:33.828+0000 7f96984497c0 -1 osd.9 0 log_to_monitors true 2024-09-13T14:44:33.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:33.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:33.912 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:44:33.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:44:33.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:33.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:34.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:35.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:35.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:35.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:44:35.316 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:44:35.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:35.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:35.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:36.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:36.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:36.650 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:44:36.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:44:36.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:36.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:37.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:38.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:38.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:38.087 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:44:38.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:44:38.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:38.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:38.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:39.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:39.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:39.433 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:44:39.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:44:39.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:39.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:39.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:40.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:40.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:40.885 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:44:40.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:44:40.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:40.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:41.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:42.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:42.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:42.209 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:44:42.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:44:42.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:42.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:42.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:43.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:43.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:43.634 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:44:43.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:44:43.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:43.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:43.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:44.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:44.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:44.972 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:44:44.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:44:44.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:44.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:45.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:46.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:46.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:46.299 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:44:46.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:44:46.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:46.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:46.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:47.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:47.747 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:44:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:44:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:47.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:48.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:49.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:49.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:49.820 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:44:49.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:44:49.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:49.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:50.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:51.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:51.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:51.380 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:44:51.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:44:51.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:51.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:51.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:52.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:52.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:52.995 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:44:52.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:44:52.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:52.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:53.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:54.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:54.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:54.699 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:44:54.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:44:54.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:54.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:55.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:56.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:56.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:56.036 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:44:56.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:44:56.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:56.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:56.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:44:57.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:44:57.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:44:57.411 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:44:57.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:44:57.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:44:57.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:44:57.750 INFO:tasks.workunit.client.0.smithi112.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/877420058,v1:127.0.0.1:6875/877420058] [v2:127.0.0.1:6876/877420058,v1:127.0.0.1:6877/877420058] exists,up 773da5cd-70ef-4087-bab9-35ba28113c89 2024-09-13T14:44:57.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:44:57.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:44:57.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:44:57.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-09-13T14:44:57.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T14:44:58.076 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' does not exist 2024-09-13T14:44:58.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T14:44:58.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T14:44:58.697 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T14:44:58.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:44:59.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T14:45:03.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-09-13T14:45:03.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:45:03.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:45:03.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:45:03.692 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:45:03.693 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:45:03.693 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:45:03.694 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:45:03.694 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:45:03.694 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:45:03.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:45:03.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:45:03.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:45:03.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:45:03.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:45:03.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:45:04.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:45:04.159 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:45:04.159 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:45:04.160 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:45:04.160 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:45:04.160 INFO:tasks.workunit.client.0.smithi112.stderr:5 2024-09-13T14:45:04.160 INFO:tasks.workunit.client.0.smithi112.stderr:6 2024-09-13T14:45:04.160 INFO:tasks.workunit.client.0.smithi112.stderr:7 2024-09-13T14:45:04.160 INFO:tasks.workunit.client.0.smithi112.stderr:8 2024-09-13T14:45:04.161 INFO:tasks.workunit.client.0.smithi112.stderr:9' 2024-09-13T14:45:04.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:45:04.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:04.161 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:45:04.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836553 2024-09-13T14:45:04.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836553 2024-09-13T14:45:04.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553' 2024-09-13T14:45:04.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:04.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:45:04.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673024 2024-09-13T14:45:04.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673024 2024-09-13T14:45:04.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673024' 2024-09-13T14:45:04.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:04.439 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:45:04.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509497 2024-09-13T14:45:04.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509497 2024-09-13T14:45:04.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673024 2-64424509497' 2024-09-13T14:45:04.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:04.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:45:04.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345974 2024-09-13T14:45:04.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345974 2024-09-13T14:45:04.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673024 2-64424509497 3-85899345974' 2024-09-13T14:45:04.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:04.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:45:04.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182444 2024-09-13T14:45:04.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182444 2024-09-13T14:45:04.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673024 2-64424509497 3-85899345974 4-107374182444' 2024-09-13T14:45:04.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:04.880 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:45:05.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018916 2024-09-13T14:45:05.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018916 2024-09-13T14:45:05.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673024 2-64424509497 3-85899345974 4-107374182444 5-128849018916' 2024-09-13T14:45:05.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:05.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:45:05.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855389 2024-09-13T14:45:05.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855389 2024-09-13T14:45:05.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673024 2-64424509497 3-85899345974 4-107374182444 5-128849018916 6-150323855389' 2024-09-13T14:45:05.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:05.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:45:05.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691865 2024-09-13T14:45:05.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691865 2024-09-13T14:45:05.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673024 2-64424509497 3-85899345974 4-107374182444 5-128849018916 6-150323855389 7-171798691865' 2024-09-13T14:45:05.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:05.308 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:45:05.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528332 2024-09-13T14:45:05.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528332 2024-09-13T14:45:05.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673024 2-64424509497 3-85899345974 4-107374182444 5-128849018916 6-150323855389 7-171798691865 8-193273528332' 2024-09-13T14:45:05.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:05.442 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:45:05.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364805 2024-09-13T14:45:05.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364805 2024-09-13T14:45:05.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673024 2-64424509497 3-85899345974 4-107374182444 5-128849018916 6-150323855389 7-171798691865 8-193273528332 9-214748364805' 2024-09-13T14:45:05.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:05.583 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836553 2024-09-13T14:45:05.583 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:05.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:45:05.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836553 2024-09-13T14:45:05.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:05.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836553 2024-09-13T14:45:05.587 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836553 2024-09-13T14:45:05.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-09-13T14:45:05.587 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:45:05.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836552 -lt 21474836553 2024-09-13T14:45:05.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:45:06.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:45:06.907 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:45:07.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836553 -lt 21474836553 2024-09-13T14:45:07.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:07.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:07.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673024 2024-09-13T14:45:07.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:45:07.240 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673024 2024-09-13T14:45:07.241 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:07.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673024 2024-09-13T14:45:07.242 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673024 2024-09-13T14:45:07.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673024' 2024-09-13T14:45:07.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:45:07.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673024 -lt 42949673024 2024-09-13T14:45:07.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:07.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509497 2024-09-13T14:45:07.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:07.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:45:07.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509497 2024-09-13T14:45:07.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:07.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509497 2024-09-13T14:45:07.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509497' 2024-09-13T14:45:07.574 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509497 2024-09-13T14:45:07.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:45:07.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509497 -lt 64424509497 2024-09-13T14:45:07.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:07.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:07.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345974 2024-09-13T14:45:07.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:45:07.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345974 2024-09-13T14:45:07.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:07.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345974 2024-09-13T14:45:07.917 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.3 seq 85899345974 2024-09-13T14:45:07.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345974' 2024-09-13T14:45:07.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:45:08.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345974 -lt 85899345974 2024-09-13T14:45:08.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:08.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:08.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182444 2024-09-13T14:45:08.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:45:08.241 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182444 2024-09-13T14:45:08.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:08.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182444 2024-09-13T14:45:08.243 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.4 seq 107374182444 2024-09-13T14:45:08.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182444' 2024-09-13T14:45:08.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:45:08.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182445 -lt 107374182444 2024-09-13T14:45:08.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:08.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018916 2024-09-13T14:45:08.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:08.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:45:08.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018916 2024-09-13T14:45:08.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:08.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018916 2024-09-13T14:45:08.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018916' 2024-09-13T14:45:08.590 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.5 seq 128849018916 2024-09-13T14:45:08.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:45:08.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018916 -lt 128849018916 2024-09-13T14:45:08.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:08.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855389 2024-09-13T14:45:08.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:08.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:45:08.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855389 2024-09-13T14:45:08.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:08.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855389 2024-09-13T14:45:08.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855389' 2024-09-13T14:45:08.906 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.6 seq 150323855389 2024-09-13T14:45:08.906 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:45:09.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855389 -lt 150323855389 2024-09-13T14:45:09.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:09.251 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691865 2024-09-13T14:45:09.251 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:09.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:45:09.253 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691865 2024-09-13T14:45:09.253 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:09.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691865 2024-09-13T14:45:09.255 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.7 seq 171798691865 2024-09-13T14:45:09.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691865' 2024-09-13T14:45:09.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:45:09.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691865 -lt 171798691865 2024-09-13T14:45:09.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:09.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528332 2024-09-13T14:45:09.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:09.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:45:09.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528332 2024-09-13T14:45:09.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:09.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528332 2024-09-13T14:45:09.606 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.8 seq 193273528332 2024-09-13T14:45:09.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528332' 2024-09-13T14:45:09.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:45:09.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528333 -lt 193273528332 2024-09-13T14:45:09.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:09.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364805 2024-09-13T14:45:09.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:09.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:45:09.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364805 2024-09-13T14:45:09.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:09.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364805 2024-09-13T14:45:09.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364805' 2024-09-13T14:45:09.915 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.9 seq 214748364805 2024-09-13T14:45:09.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:45:10.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364806 -lt 214748364805 2024-09-13T14:45:10.241 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:45:10.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:45:10.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:45:10.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T14:45:10.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:45:10.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:45:10.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:45:10.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:45:10.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:45:10.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:45:10.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:45:10.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:45:10.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:45:10.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:45:10.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:45:11.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T14:45:11.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:45:11.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:45:11.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-09-13T14:45:11.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-13T14:45:11.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-13T14:45:11.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-13T14:45:11.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-13T14:45:11.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-09-13T14:45:11.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-13T14:45:11.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-13T14:45:12.501 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'ecpool' created 2024-09-13T14:45:12.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:45:13.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-13T14:45:13.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-13T14:45:13.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:45:13.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:45:13.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:45:13.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:45:13.525 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:45:13.525 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:45:13.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:45:13.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:45:13.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:45:13.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:45:13.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:45:13.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:45:13.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:45:13.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:45:13.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:45:13.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:45:13.990 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:45:13.990 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:45:13.990 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:45:13.990 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:45:13.990 INFO:tasks.workunit.client.0.smithi112.stderr:5 2024-09-13T14:45:13.990 INFO:tasks.workunit.client.0.smithi112.stderr:6 2024-09-13T14:45:13.990 INFO:tasks.workunit.client.0.smithi112.stderr:7 2024-09-13T14:45:13.990 INFO:tasks.workunit.client.0.smithi112.stderr:8 2024-09-13T14:45:13.990 INFO:tasks.workunit.client.0.smithi112.stderr:9' 2024-09-13T14:45:13.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:45:13.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:13.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:45:14.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836556 2024-09-13T14:45:14.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836556 2024-09-13T14:45:14.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556' 2024-09-13T14:45:14.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:14.167 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:45:14.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673027 2024-09-13T14:45:14.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673027 2024-09-13T14:45:14.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673027' 2024-09-13T14:45:14.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:14.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:45:14.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509500 2024-09-13T14:45:14.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509500 2024-09-13T14:45:14.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673027 2-64424509500' 2024-09-13T14:45:14.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:14.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:45:14.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345977 2024-09-13T14:45:14.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345977 2024-09-13T14:45:14.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673027 2-64424509500 3-85899345977' 2024-09-13T14:45:14.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:14.811 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:45:14.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182447 2024-09-13T14:45:14.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182447 2024-09-13T14:45:14.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673027 2-64424509500 3-85899345977 4-107374182447' 2024-09-13T14:45:14.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:14.956 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:45:15.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018919 2024-09-13T14:45:15.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018919 2024-09-13T14:45:15.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673027 2-64424509500 3-85899345977 4-107374182447 5-128849018919' 2024-09-13T14:45:15.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:15.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:45:15.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855392 2024-09-13T14:45:15.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855392 2024-09-13T14:45:15.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673027 2-64424509500 3-85899345977 4-107374182447 5-128849018919 6-150323855392' 2024-09-13T14:45:15.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:15.249 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:45:15.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691868 2024-09-13T14:45:15.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691868 2024-09-13T14:45:15.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673027 2-64424509500 3-85899345977 4-107374182447 5-128849018919 6-150323855392 7-171798691868' 2024-09-13T14:45:15.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:15.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:45:15.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528335 2024-09-13T14:45:15.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528335 2024-09-13T14:45:15.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673027 2-64424509500 3-85899345977 4-107374182447 5-128849018919 6-150323855392 7-171798691868 8-193273528335' 2024-09-13T14:45:15.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:15.536 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:45:15.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364808 2024-09-13T14:45:15.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364808 2024-09-13T14:45:15.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673027 2-64424509500 3-85899345977 4-107374182447 5-128849018919 6-150323855392 7-171798691868 8-193273528335 9-214748364808' 2024-09-13T14:45:15.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:15.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836556 2024-09-13T14:45:15.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:15.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:45:15.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836556 2024-09-13T14:45:15.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:15.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836556 2024-09-13T14:45:15.683 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836556 2024-09-13T14:45:15.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-09-13T14:45:15.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:45:15.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836556 -lt 21474836556 2024-09-13T14:45:15.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:15.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673027 2024-09-13T14:45:15.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:45:15.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673027 2024-09-13T14:45:15.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:15.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673027 2024-09-13T14:45:15.995 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673027 2024-09-13T14:45:15.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673027' 2024-09-13T14:45:15.996 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:45:16.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673027 -lt 42949673027 2024-09-13T14:45:16.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:16.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509500 2024-09-13T14:45:16.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:16.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:45:16.361 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509500 2024-09-13T14:45:16.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:16.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509500 2024-09-13T14:45:16.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509500' 2024-09-13T14:45:16.364 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509500 2024-09-13T14:45:16.364 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:45:16.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509500 -lt 64424509500 2024-09-13T14:45:16.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:16.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:16.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345977 2024-09-13T14:45:16.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:45:16.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345977 2024-09-13T14:45:16.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:16.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345977 2024-09-13T14:45:16.723 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.3 seq 85899345977 2024-09-13T14:45:16.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345977' 2024-09-13T14:45:16.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:45:17.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345977 -lt 85899345977 2024-09-13T14:45:17.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:17.065 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182447 2024-09-13T14:45:17.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:17.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:45:17.068 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182447 2024-09-13T14:45:17.068 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:17.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182447 2024-09-13T14:45:17.069 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.4 seq 107374182447 2024-09-13T14:45:17.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182447' 2024-09-13T14:45:17.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:45:17.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182447 -lt 107374182447 2024-09-13T14:45:17.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:17.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018919 2024-09-13T14:45:17.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:17.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:45:17.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018919 2024-09-13T14:45:17.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:17.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018919 2024-09-13T14:45:17.410 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.5 seq 128849018919 2024-09-13T14:45:17.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018919' 2024-09-13T14:45:17.410 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:45:17.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018919 -lt 128849018919 2024-09-13T14:45:17.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:17.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:17.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855392 2024-09-13T14:45:17.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:45:17.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855392 2024-09-13T14:45:17.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:17.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855392 2024-09-13T14:45:17.769 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.6 seq 150323855392 2024-09-13T14:45:17.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855392' 2024-09-13T14:45:17.769 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:45:18.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855392 -lt 150323855392 2024-09-13T14:45:18.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:18.120 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691868 2024-09-13T14:45:18.120 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:18.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:45:18.122 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691868 2024-09-13T14:45:18.122 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:18.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691868 2024-09-13T14:45:18.123 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.7 seq 171798691868 2024-09-13T14:45:18.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691868' 2024-09-13T14:45:18.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:45:18.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691868 -lt 171798691868 2024-09-13T14:45:18.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:18.498 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528335 2024-09-13T14:45:18.498 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:18.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:45:18.501 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528335 2024-09-13T14:45:18.501 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:18.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528335 2024-09-13T14:45:18.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528335' 2024-09-13T14:45:18.503 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.8 seq 193273528335 2024-09-13T14:45:18.503 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:45:18.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528336 -lt 193273528335 2024-09-13T14:45:18.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:18.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364808 2024-09-13T14:45:18.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:18.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:45:18.840 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364808 2024-09-13T14:45:18.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:18.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364808 2024-09-13T14:45:18.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364808' 2024-09-13T14:45:18.842 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.9 seq 214748364808 2024-09-13T14:45:18.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:45:19.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364809 -lt 214748364808 2024-09-13T14:45:19.164 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:45:19.165 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:45:19.165 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:45:19.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:45:19.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:45:19.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:45:19.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:45:19.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:45:19.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:45:19.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:45:19.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:45:19.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:45:19.895 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:45:19.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:45:19.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:45:20.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:45:20.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:45:20.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:45:20.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-13T14:45:20.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-13T14:45:20.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-13T14:45:20.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-13T14:45:20.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-13T14:45:20.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T14:45:20.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T14:45:20.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-13T14:45:20.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T14:45:20.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T14:45:20.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T14:45:20.751 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T14:45:20.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T14:45:21.312 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T14:45:21.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T14:45:21.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T14:45:21.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-13T14:45:21.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-13T14:45:21.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-13T14:45:21.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:45:21.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:45:21.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:45:22.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-13T14:45:22.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-13T14:45:22.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-13T14:45:22.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-13T14:45:22.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-13T14:45:22.170 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T14:45:22.170 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:45:22.540 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-13T14:45:22.540 INFO:tasks.workunit.client.0.smithi112.stderr:5 2024-09-13T14:45:22.540 INFO:tasks.workunit.client.0.smithi112.stderr:9 2024-09-13T14:45:22.540 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:45:22.540 INFO:tasks.workunit.client.0.smithi112.stderr:6 2024-09-13T14:45:22.540 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:45:22.540 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:45:22.540 INFO:tasks.workunit.client.0.smithi112.stderr:7' 2024-09-13T14:45:22.540 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 6 2 1 7 2024-09-13T14:45:22.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-09-13T14:45:22.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-13T14:45:22.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-09-13T14:45:22.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-09-13T14:45:22.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-13T14:45:22.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-13T14:45:22.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-13T14:45:22.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-13T14:45:22.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:45:22.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:45:22.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:45:22.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:45:22.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:45:22.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:45:22.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:45:22.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:45:22.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:45:22.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:45:22.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:45:22.543 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:45:22.543 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:45:22.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:45:22.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:45:22.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:45:22.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:45:22.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:45:22.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:45:22.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:45:22.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:45:22.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:45:22.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:45:22.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-13T14:45:24.981 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-13T14:45:25.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:45:25.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:45:25.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:45:25.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:45:25.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:45:25.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:45:25.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:45:25.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:45:25.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:45:25.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:45:25.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:45:25.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:45:25.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:45:25.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:45:25.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:45:25.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:45:25.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:45:25.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:45:25.523 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:45:25.523 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:45:25.523 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:45:25.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:45:25.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:45:25.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:45:25.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:45:25.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:45:25.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:45:25.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:45:25.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:45:25.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:45:25.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:45:25.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:45:25.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:45:25.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:45:25.525 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.3 2024-09-13T14:45:25.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:45:25.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:45:25.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:45:25.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:45:25.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:45:25.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:45:25.561 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:45:25.558+0000 7fc1836487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:45:25.564 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:45:25.562+0000 7fc1836487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:45:25.565 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:45:25.562+0000 7fc1836487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:45:25.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:45:25.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:45:25.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:45:25.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:45:25.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:45:25.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:45:25.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:45:25.864 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:45:25.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:45:25.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:45:26.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:45:26.621 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:45:26.618+0000 7fc1836487c0 -1 Falling back to public interface 2024-09-13T14:45:27.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:45:27.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:45:27.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:45:27.195 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:45:27.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:45:27.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:45:27.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:45:28.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:45:28.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:45:28.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:45:28.532 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:45:28.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:45:28.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:45:28.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:45:29.448 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:45:29.446+0000 7fc1836487c0 -1 osd.3 69 log_to_monitors true 2024-09-13T14:45:29.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:45:29.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:45:29.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:45:29.868 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:45:29.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:45:29.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:45:30.406 INFO:tasks.workunit.client.0.smithi112.stderr:osd.3 up in weight 1 up_from 73 up_thru 65 down_at 70 last_clean_interval [20,69) [v2:127.0.0.1:6826/4223012903,v1:127.0.0.1:6827/4223012903] [v2:127.0.0.1:6828/4223012903,v1:127.0.0.1:6829/4223012903] exists,up 240beeea-9a32-48d8-9b95-819a8b7fdfbd 2024-09-13T14:45:30.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:45:30.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:45:30.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:45:30.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:45:30.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:45:30.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:45:30.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:45:30.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:45:30.407 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:45:30.407 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:45:30.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:45:30.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:45:30.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:45:30.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:45:30.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:45:30.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:45:30.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:45:30.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:45:30.521 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:45:30.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:45:30.892 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:45:30.893 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:45:30.893 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:45:30.893 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:45:30.893 INFO:tasks.workunit.client.0.smithi112.stderr:5 2024-09-13T14:45:30.893 INFO:tasks.workunit.client.0.smithi112.stderr:6 2024-09-13T14:45:30.893 INFO:tasks.workunit.client.0.smithi112.stderr:7 2024-09-13T14:45:30.893 INFO:tasks.workunit.client.0.smithi112.stderr:8 2024-09-13T14:45:30.893 INFO:tasks.workunit.client.0.smithi112.stderr:9' 2024-09-13T14:45:30.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:45:30.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:30.894 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:45:31.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836560 2024-09-13T14:45:31.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836560 2024-09-13T14:45:31.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560' 2024-09-13T14:45:31.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:31.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:45:31.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673031 2024-09-13T14:45:31.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673031 2024-09-13T14:45:31.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673031' 2024-09-13T14:45:31.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:31.334 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:45:31.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509505 2024-09-13T14:45:31.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509505 2024-09-13T14:45:31.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673031 2-64424509505' 2024-09-13T14:45:31.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:31.477 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:45:31.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612610 2024-09-13T14:45:31.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612610 2024-09-13T14:45:31.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673031 2-64424509505 3-313532612610' 2024-09-13T14:45:31.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:31.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:45:31.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182451 2024-09-13T14:45:31.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182451 2024-09-13T14:45:31.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673031 2-64424509505 3-313532612610 4-107374182451' 2024-09-13T14:45:31.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:31.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:45:31.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018923 2024-09-13T14:45:31.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018923 2024-09-13T14:45:31.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673031 2-64424509505 3-313532612610 4-107374182451 5-128849018923' 2024-09-13T14:45:31.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:31.883 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:45:32.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855396 2024-09-13T14:45:32.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855396 2024-09-13T14:45:32.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673031 2-64424509505 3-313532612610 4-107374182451 5-128849018923 6-150323855396' 2024-09-13T14:45:32.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:32.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:45:32.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691872 2024-09-13T14:45:32.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691872 2024-09-13T14:45:32.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673031 2-64424509505 3-313532612610 4-107374182451 5-128849018923 6-150323855396 7-171798691872' 2024-09-13T14:45:32.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:32.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:45:32.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528340 2024-09-13T14:45:32.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528340 2024-09-13T14:45:32.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673031 2-64424509505 3-313532612610 4-107374182451 5-128849018923 6-150323855396 7-171798691872 8-193273528340' 2024-09-13T14:45:32.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:32.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:45:32.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364813 2024-09-13T14:45:32.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364813 2024-09-13T14:45:32.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673031 2-64424509505 3-313532612610 4-107374182451 5-128849018923 6-150323855396 7-171798691872 8-193273528340 9-214748364813' 2024-09-13T14:45:32.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:32.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836560 2024-09-13T14:45:32.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:32.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:45:32.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836560 2024-09-13T14:45:32.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:32.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836560 2024-09-13T14:45:32.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836560' 2024-09-13T14:45:32.457 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836560 2024-09-13T14:45:32.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:45:32.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836560 -lt 21474836560 2024-09-13T14:45:32.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:32.776 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:32.776 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673031 2024-09-13T14:45:32.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:45:32.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673031 2024-09-13T14:45:32.779 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:32.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673031 2024-09-13T14:45:32.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2024-09-13T14:45:32.780 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949673031 2024-09-13T14:45:32.780 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:45:33.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673031 -lt 42949673031 2024-09-13T14:45:33.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:33.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509505 2024-09-13T14:45:33.088 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:33.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:45:33.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509505 2024-09-13T14:45:33.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:33.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509505 2024-09-13T14:45:33.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2024-09-13T14:45:33.091 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509505 2024-09-13T14:45:33.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:45:33.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509505 -lt 64424509505 2024-09-13T14:45:33.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:33.410 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612610 2024-09-13T14:45:33.410 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:33.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:45:33.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612610 2024-09-13T14:45:33.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:33.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612610 2024-09-13T14:45:33.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612610' 2024-09-13T14:45:33.415 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 313532612610 2024-09-13T14:45:33.415 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:45:33.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612610 -lt 313532612610 2024-09-13T14:45:33.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:33.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182451 2024-09-13T14:45:33.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:33.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:45:33.780 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182451 2024-09-13T14:45:33.780 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:33.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182451 2024-09-13T14:45:33.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2024-09-13T14:45:33.782 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.4 seq 107374182451 2024-09-13T14:45:33.782 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:45:34.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182452 -lt 107374182451 2024-09-13T14:45:34.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:34.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018923 2024-09-13T14:45:34.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:34.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:45:34.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018923 2024-09-13T14:45:34.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:34.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018923 2024-09-13T14:45:34.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018923' 2024-09-13T14:45:34.110 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.5 seq 128849018923 2024-09-13T14:45:34.110 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:45:34.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018924 -lt 128849018923 2024-09-13T14:45:34.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:34.477 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855396 2024-09-13T14:45:34.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:34.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:45:34.480 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855396 2024-09-13T14:45:34.480 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:34.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855396 2024-09-13T14:45:34.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855396' 2024-09-13T14:45:34.482 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.6 seq 150323855396 2024-09-13T14:45:34.482 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:45:34.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855397 -lt 150323855396 2024-09-13T14:45:34.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:34.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691872 2024-09-13T14:45:34.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:45:34.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691872 2024-09-13T14:45:34.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:34.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691872 2024-09-13T14:45:34.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691872' 2024-09-13T14:45:34.829 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.7 seq 171798691872 2024-09-13T14:45:34.829 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:45:35.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691872 -lt 171798691872 2024-09-13T14:45:35.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:35.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528340 2024-09-13T14:45:35.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:35.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:45:35.155 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528340 2024-09-13T14:45:35.155 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:35.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528340 2024-09-13T14:45:35.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528340' 2024-09-13T14:45:35.156 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.8 seq 193273528340 2024-09-13T14:45:35.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:45:35.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528340 -lt 193273528340 2024-09-13T14:45:35.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:35.556 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364813 2024-09-13T14:45:35.556 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:35.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:45:35.559 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364813 2024-09-13T14:45:35.559 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:35.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364813 2024-09-13T14:45:35.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364813' 2024-09-13T14:45:35.561 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.9 seq 214748364813 2024-09-13T14:45:35.561 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:45:35.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364813 -lt 214748364813 2024-09-13T14:45:35.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:45:35.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:45:35.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:45:36.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:45:36.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:45:36.307 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:45:36.307 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:45:36.307 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:45:36.307 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:45:36.307 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:45:36.307 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:45:36.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:45:36.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:45:36.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:45:36.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:45:37.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:45:37.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:45:37.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:45:37.036 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-13T14:45:37.036 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:45:37.036 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:45:37.036 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:45:37.036 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:45:37.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-13T14:45:37.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-13T14:45:37.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:45:37.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:45:37.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:45:37.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:45:37.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:45:37.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:45:37.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:45:12.392472+0000 2024-09-13T14:45:37.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:45:38.045 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:45:38.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:45:12.392472+0000 2024-09-13T14:45:38.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:45:38.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:45:12.392472+0000 2024-09-13T14:45:38.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:45:38.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:45:38.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:45:38.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:45:38.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:45:38.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:45:38.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:45:38.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:45:38.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:45:12.392472+0000 '>' 2024-09-13T14:45:12.392472+0000 2024-09-13T14:45:38.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:45:39.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:45:39.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:45:39.390 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:45:39.390 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:45:39.390 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:45:39.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:45:39.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:45:39.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:45:12.392472+0000 '>' 2024-09-13T14:45:12.392472+0000 2024-09-13T14:45:39.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:45:40.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:45:40.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:45:40.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:45:40.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:45:40.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:45:40.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:45:40.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:45:41.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:45:38.454385+0000 '>' 2024-09-13T14:45:12.392472+0000 2024-09-13T14:45:41.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:45:41.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:45:43.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:45:43.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:45:43.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:45:43.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-13T14:45:43.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:45:43.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:45:43.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:45:43.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:45:43.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-13T14:45:43.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:45:43.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-13T14:45:43.077 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:45:43.077 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:45:43.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:45:43.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:45:43.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:45:43.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:45:43.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:45:43.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:45:43.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:45:43.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-13T14:45:43.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:45:43.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:45:43.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-13T14:45:44.159 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:45:44.159 INFO:tasks.workunit.client.0.smithi112.stdout:hinfo_key 2024-09-13T14:45:44.159 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:45:44.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-13T14:45:44.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:45:44.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:45:44.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-13T14:45:44.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:45:44.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:45:44.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:45:44.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:45:44.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:45:44.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:45:44.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-13T14:45:44.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-13T14:45:44.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:45:44.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:45:44.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:45:44.455 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:45:44.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:45:44.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:45:44.457 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:45:44.457 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:45:44.457 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:45:44.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:45:44.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:45:44.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:45:44.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:45:44.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:45:44.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:45:44.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:45:44.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:45:44.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:45:44.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:45:44.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:45:44.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-13T14:45:44.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-13T14:45:44.458 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.9 2024-09-13T14:45:44.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:45:44.459 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-13T14:45:44.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-13T14:45:44.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:45:44.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:45:44.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:45:44.492 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:45:44.490+0000 7fc6e83087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:45:44.495 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:45:44.494+0000 7fc6e83087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:45:44.497 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:45:44.494+0000 7fc6e83087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:45:44.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-13T14:45:44.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:45:44.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-13T14:45:44.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:45:44.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:45:44.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:45:44.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:45:44.825 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:45:44.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:45:44.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:45:45.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:45:45.809 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:45:45.806+0000 7fc6e83087c0 -1 Falling back to public interface 2024-09-13T14:45:46.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:45:46.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:45:46.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:45:46.163 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:45:46.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:45:46.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:45:46.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:45:47.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:45:47.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:45:47.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:45:47.500 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:45:47.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:45:47.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:45:47.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:45:48.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:45:48.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:45:48.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:45:48.837 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:45:48.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:45:48.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:45:48.984 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:45:48.982+0000 7fc6e83087c0 -1 osd.9 74 log_to_monitors true 2024-09-13T14:45:49.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:45:50.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:45:50.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:45:50.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:45:50.203 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:45:50.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:45:50.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:45:50.869 INFO:tasks.workunit.client.0.smithi112.stderr:osd.9 up in weight 1 up_from 78 up_thru 0 down_at 75 last_clean_interval [50,74) [v2:127.0.0.1:6874/748137614,v1:127.0.0.1:6875/748137614] [v2:127.0.0.1:6876/748137614,v1:127.0.0.1:6877/748137614] exists,up 773da5cd-70ef-4087-bab9-35ba28113c89 2024-09-13T14:45:50.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:45:50.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:45:50.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:45:50.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:45:50.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:45:50.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:45:50.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:45:50.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:45:50.871 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:45:50.871 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:45:50.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:45:50.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:45:50.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:45:50.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:45:50.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:45:50.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:45:50.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:45:50.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:45:50.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:45:51.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:45:51.421 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:45:51.421 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:45:51.421 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:45:51.421 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:45:51.421 INFO:tasks.workunit.client.0.smithi112.stderr:5 2024-09-13T14:45:51.421 INFO:tasks.workunit.client.0.smithi112.stderr:6 2024-09-13T14:45:51.421 INFO:tasks.workunit.client.0.smithi112.stderr:7 2024-09-13T14:45:51.421 INFO:tasks.workunit.client.0.smithi112.stderr:8 2024-09-13T14:45:51.421 INFO:tasks.workunit.client.0.smithi112.stderr:9' 2024-09-13T14:45:51.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:45:51.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:51.421 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:45:51.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836565 2024-09-13T14:45:51.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836565 2024-09-13T14:45:51.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565' 2024-09-13T14:45:51.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:51.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:45:51.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673036 2024-09-13T14:45:51.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673036 2024-09-13T14:45:51.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673036' 2024-09-13T14:45:51.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:51.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:45:51.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509510 2024-09-13T14:45:51.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509510 2024-09-13T14:45:51.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509510' 2024-09-13T14:45:51.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:51.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:45:52.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612615 2024-09-13T14:45:52.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612615 2024-09-13T14:45:52.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509510 3-313532612615' 2024-09-13T14:45:52.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:52.091 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:45:52.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182457 2024-09-13T14:45:52.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182457 2024-09-13T14:45:52.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509510 3-313532612615 4-107374182457' 2024-09-13T14:45:52.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:52.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:45:52.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018928 2024-09-13T14:45:52.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018928 2024-09-13T14:45:52.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509510 3-313532612615 4-107374182457 5-128849018928' 2024-09-13T14:45:52.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:52.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:45:52.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855401 2024-09-13T14:45:52.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855401 2024-09-13T14:45:52.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509510 3-313532612615 4-107374182457 5-128849018928 6-150323855401' 2024-09-13T14:45:52.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:52.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:45:52.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691877 2024-09-13T14:45:52.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691877 2024-09-13T14:45:52.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509510 3-313532612615 4-107374182457 5-128849018928 6-150323855401 7-171798691877' 2024-09-13T14:45:52.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:52.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:45:52.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528345 2024-09-13T14:45:52.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528345 2024-09-13T14:45:52.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509510 3-313532612615 4-107374182457 5-128849018928 6-150323855401 7-171798691877 8-193273528345' 2024-09-13T14:45:52.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:45:52.822 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:45:52.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449090 2024-09-13T14:45:52.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449090 2024-09-13T14:45:52.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673036 2-64424509510 3-313532612615 4-107374182457 5-128849018928 6-150323855401 7-171798691877 8-193273528345 9-335007449090' 2024-09-13T14:45:52.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:52.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836565 2024-09-13T14:45:52.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:52.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:45:52.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836565 2024-09-13T14:45:52.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:52.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836565 2024-09-13T14:45:52.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-09-13T14:45:52.968 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836565 2024-09-13T14:45:52.969 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:45:53.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836565 -lt 21474836565 2024-09-13T14:45:53.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:53.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673036 2024-09-13T14:45:53.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:53.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:45:53.295 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673036 2024-09-13T14:45:53.295 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:53.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673036 2024-09-13T14:45:53.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673036' 2024-09-13T14:45:53.297 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949673036 2024-09-13T14:45:53.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:45:53.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673036 -lt 42949673036 2024-09-13T14:45:53.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:53.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509510 2024-09-13T14:45:53.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:53.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:45:53.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509510 2024-09-13T14:45:53.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:53.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509510 2024-09-13T14:45:53.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-09-13T14:45:53.645 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509510 2024-09-13T14:45:53.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:45:53.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509510 -lt 64424509510 2024-09-13T14:45:53.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:53.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612615 2024-09-13T14:45:53.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:53.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:45:53.999 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612615 2024-09-13T14:45:53.999 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:54.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612615 2024-09-13T14:45:54.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612615' 2024-09-13T14:45:54.000 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 313532612615 2024-09-13T14:45:54.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:45:54.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612615 -lt 313532612615 2024-09-13T14:45:54.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:54.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:54.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182457 2024-09-13T14:45:54.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:45:54.334 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182457 2024-09-13T14:45:54.334 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:54.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182457 2024-09-13T14:45:54.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182457' 2024-09-13T14:45:54.336 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.4 seq 107374182457 2024-09-13T14:45:54.336 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:45:54.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182457 -lt 107374182457 2024-09-13T14:45:54.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:54.665 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018928 2024-09-13T14:45:54.665 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:54.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:45:54.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018928 2024-09-13T14:45:54.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:54.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018928 2024-09-13T14:45:54.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018928' 2024-09-13T14:45:54.670 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.5 seq 128849018928 2024-09-13T14:45:54.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:45:54.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018929 -lt 128849018928 2024-09-13T14:45:54.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:54.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855401 2024-09-13T14:45:54.998 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:54.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:45:55.000 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855401 2024-09-13T14:45:55.000 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:55.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855401 2024-09-13T14:45:55.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855401' 2024-09-13T14:45:55.001 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.6 seq 150323855401 2024-09-13T14:45:55.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:45:55.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855402 -lt 150323855401 2024-09-13T14:45:55.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:55.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691877 2024-09-13T14:45:55.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:55.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:45:55.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691877 2024-09-13T14:45:55.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:55.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691877 2024-09-13T14:45:55.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691877' 2024-09-13T14:45:55.331 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.7 seq 171798691877 2024-09-13T14:45:55.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:45:55.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691877 -lt 171798691877 2024-09-13T14:45:55.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:55.669 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528345 2024-09-13T14:45:55.669 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:55.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:45:55.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528345 2024-09-13T14:45:55.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:55.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528345 2024-09-13T14:45:55.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528345' 2024-09-13T14:45:55.673 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.8 seq 193273528345 2024-09-13T14:45:55.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:45:56.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528345 -lt 193273528345 2024-09-13T14:45:56.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:45:56.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-335007449090 2024-09-13T14:45:56.011 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:45:56.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:45:56.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-335007449090 2024-09-13T14:45:56.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:45:56.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449090 2024-09-13T14:45:56.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 335007449090' 2024-09-13T14:45:56.014 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.9 seq 335007449090 2024-09-13T14:45:56.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:45:56.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449091 -lt 335007449090 2024-09-13T14:45:56.372 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:45:56.372 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:45:56.372 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:45:56.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:45:56.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:45:56.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:45:56.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:45:56.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:45:56.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:45:56.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:45:56.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:45:57.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:45:57.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:45:57.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:45:57.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:45:57.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:45:57.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:45:57.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:45:57.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:45:57.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:45:57.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-09-13T14:45:57.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-13T14:45:57.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-13T14:45:57.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-09-13T14:45:57.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:45:57.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:45:57.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:45:57.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-13T14:45:57.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:45:57.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:45:57.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:45:57.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:45:57.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-13T14:45:57.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:45:57.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-13T14:45:57.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:45:57.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:45:57.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:45:57.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:45:57.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:45:57.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:45:57.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:45:57.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:45:57.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:45:57.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-13T14:45:57.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:45:57.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:45:57.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-13T14:45:59.784 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-13T14:46:00.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-13T14:46:00.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:46:00.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:46:00.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-13T14:46:00.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:46:00.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:46:00.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:46:00.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:46:00.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:46:00.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:46:00.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-13T14:46:00.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-13T14:46:00.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:46:00.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:46:00.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:46:00.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:46:00.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:46:00.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:46:00.321 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:46:00.321 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:46:00.321 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:46:00.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:46:00.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:46:00.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:46:00.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:46:00.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:46:00.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:46:00.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:46:00.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:46:00.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:46:00.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:46:00.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:46:00.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-13T14:46:00.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-13T14:46:00.323 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.5 2024-09-13T14:46:00.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:46:00.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-13T14:46:00.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-13T14:46:00.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:46:00.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:46:00.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:46:00.356 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:46:00.353+0000 7eff9ab487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:46:00.359 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:46:00.357+0000 7eff9ab487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:46:00.361 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:46:00.357+0000 7eff9ab487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:46:00.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-13T14:46:00.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:46:00.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-13T14:46:00.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:46:00.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:46:00.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:00.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:46:00.673 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:46:00.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:00.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:46:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:01.426 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:46:01.421+0000 7eff9ab487c0 -1 Falling back to public interface 2024-09-13T14:46:02.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:02.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:02.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:46:02.004 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:46:02.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:02.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:46:02.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:03.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:03.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:03.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:46:03.342 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:46:03.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:03.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:46:03.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:04.153 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:46:04.149+0000 7eff9ab487c0 -1 osd.5 80 log_to_monitors true 2024-09-13T14:46:04.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:04.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:04.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:46:04.669 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:46:04.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:04.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:46:05.375 INFO:tasks.workunit.client.0.smithi112.stderr:osd.5 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [30,80) [v2:127.0.0.1:6842/3084529851,v1:127.0.0.1:6843/3084529851] [v2:127.0.0.1:6844/3084529851,v1:127.0.0.1:6845/3084529851] exists,up ac5b44cf-f8cb-4d05-8d84-3858f28ea531 2024-09-13T14:46:05.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:46:05.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:46:05.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:46:05.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:46:05.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:46:05.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:46:05.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:46:05.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:46:05.377 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:46:05.377 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:46:05.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:46:05.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:46:05.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:46:05.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:46:05.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:46:05.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:46:05.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:46:05.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:46:05.486 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:46:05.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:46:05.822 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:46:05.823 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:46:05.823 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:46:05.823 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:46:05.823 INFO:tasks.workunit.client.0.smithi112.stderr:5 2024-09-13T14:46:05.823 INFO:tasks.workunit.client.0.smithi112.stderr:6 2024-09-13T14:46:05.823 INFO:tasks.workunit.client.0.smithi112.stderr:7 2024-09-13T14:46:05.823 INFO:tasks.workunit.client.0.smithi112.stderr:8 2024-09-13T14:46:05.823 INFO:tasks.workunit.client.0.smithi112.stderr:9' 2024-09-13T14:46:05.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:46:05.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:05.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:46:06.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836569 2024-09-13T14:46:06.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836569 2024-09-13T14:46:06.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569' 2024-09-13T14:46:06.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:06.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:46:06.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673040 2024-09-13T14:46:06.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673040 2024-09-13T14:46:06.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673040' 2024-09-13T14:46:06.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:06.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:46:06.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509514 2024-09-13T14:46:06.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509514 2024-09-13T14:46:06.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514' 2024-09-13T14:46:06.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:06.821 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:46:06.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612619 2024-09-13T14:46:06.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612619 2024-09-13T14:46:06.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514 3-313532612619' 2024-09-13T14:46:06.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:06.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:46:07.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182461 2024-09-13T14:46:07.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182461 2024-09-13T14:46:07.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514 3-313532612619 4-107374182461' 2024-09-13T14:46:07.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:07.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:46:07.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252866 2024-09-13T14:46:07.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252866 2024-09-13T14:46:07.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514 3-313532612619 4-107374182461 5-360777252866' 2024-09-13T14:46:07.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:07.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:46:07.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855405 2024-09-13T14:46:07.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855405 2024-09-13T14:46:07.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514 3-313532612619 4-107374182461 5-360777252866 6-150323855405' 2024-09-13T14:46:07.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:07.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:46:07.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691881 2024-09-13T14:46:07.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691881 2024-09-13T14:46:07.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514 3-313532612619 4-107374182461 5-360777252866 6-150323855405 7-171798691881' 2024-09-13T14:46:07.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:07.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:46:07.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528349 2024-09-13T14:46:07.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528349 2024-09-13T14:46:07.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514 3-313532612619 4-107374182461 5-360777252866 6-150323855405 7-171798691881 8-193273528349' 2024-09-13T14:46:07.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:07.689 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:46:07.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449094 2024-09-13T14:46:07.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449094 2024-09-13T14:46:07.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514 3-313532612619 4-107374182461 5-360777252866 6-150323855405 7-171798691881 8-193273528349 9-335007449094' 2024-09-13T14:46:07.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:07.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836569 2024-09-13T14:46:07.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:07.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:46:07.834 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836569 2024-09-13T14:46:07.834 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:07.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836569 2024-09-13T14:46:07.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-09-13T14:46:07.835 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836569 2024-09-13T14:46:07.836 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:46:08.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836568 -lt 21474836569 2024-09-13T14:46:08.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:46:09.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:46:09.167 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:46:09.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836570 -lt 21474836569 2024-09-13T14:46:09.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:09.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673040 2024-09-13T14:46:09.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:09.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:46:09.498 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673040 2024-09-13T14:46:09.499 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:09.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673040 2024-09-13T14:46:09.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-09-13T14:46:09.500 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949673040 2024-09-13T14:46:09.500 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:46:09.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673040 -lt 42949673040 2024-09-13T14:46:09.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:09.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:09.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509514 2024-09-13T14:46:09.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:46:09.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509514 2024-09-13T14:46:09.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:09.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509514 2024-09-13T14:46:09.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-09-13T14:46:09.827 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509514 2024-09-13T14:46:09.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:46:10.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509514 -lt 64424509514 2024-09-13T14:46:10.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:10.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612619 2024-09-13T14:46:10.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:10.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:46:10.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612619 2024-09-13T14:46:10.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:10.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612619 2024-09-13T14:46:10.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612619' 2024-09-13T14:46:10.161 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 313532612619 2024-09-13T14:46:10.162 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:46:10.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612620 -lt 313532612619 2024-09-13T14:46:10.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:10.499 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182461 2024-09-13T14:46:10.499 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:10.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:46:10.500 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182461 2024-09-13T14:46:10.500 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:10.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182461 2024-09-13T14:46:10.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182461' 2024-09-13T14:46:10.502 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.4 seq 107374182461 2024-09-13T14:46:10.502 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:46:10.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182461 -lt 107374182461 2024-09-13T14:46:10.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:10.835 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-360777252866 2024-09-13T14:46:10.836 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:10.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:46:10.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:10.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-360777252866 2024-09-13T14:46:10.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252866 2024-09-13T14:46:10.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 360777252866' 2024-09-13T14:46:10.839 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.5 seq 360777252866 2024-09-13T14:46:10.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:46:11.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252867 -lt 360777252866 2024-09-13T14:46:11.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:11.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855405 2024-09-13T14:46:11.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:11.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:46:11.142 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855405 2024-09-13T14:46:11.142 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:11.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855405 2024-09-13T14:46:11.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855405' 2024-09-13T14:46:11.144 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.6 seq 150323855405 2024-09-13T14:46:11.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:46:11.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855406 -lt 150323855405 2024-09-13T14:46:11.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:11.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:11.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691881 2024-09-13T14:46:11.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:46:11.467 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691881 2024-09-13T14:46:11.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:11.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691881 2024-09-13T14:46:11.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691881' 2024-09-13T14:46:11.469 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.7 seq 171798691881 2024-09-13T14:46:11.469 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:46:11.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691881 -lt 171798691881 2024-09-13T14:46:11.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:11.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528349 2024-09-13T14:46:11.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:11.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:46:11.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528349 2024-09-13T14:46:11.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:11.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528349 2024-09-13T14:46:11.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528349' 2024-09-13T14:46:11.817 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.8 seq 193273528349 2024-09-13T14:46:11.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:46:12.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528349 -lt 193273528349 2024-09-13T14:46:12.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:12.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-335007449094 2024-09-13T14:46:12.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:12.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:46:12.155 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-335007449094 2024-09-13T14:46:12.155 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:12.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449094 2024-09-13T14:46:12.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 335007449094' 2024-09-13T14:46:12.157 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.9 seq 335007449094 2024-09-13T14:46:12.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:46:12.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449095 -lt 335007449094 2024-09-13T14:46:12.512 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:46:12.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:46:12.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:46:12.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:46:12.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:46:12.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:46:12.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:46:12.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:46:12.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:46:12.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:46:12.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:46:13.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:46:13.253 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:46:13.253 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:46:13.253 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:46:13.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:46:13.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:46:13.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:46:13.631 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-13T14:46:13.631 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:46:13.631 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:46:13.631 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:46:13.632 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:46:13.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-13T14:46:13.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-13T14:46:13.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:46:13.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:46:13.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:46:13.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:46:13.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:46:13.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:46:14.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:45:38.454385+0000 2024-09-13T14:46:14.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:46:14.582 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:46:14.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:45:38.454385+0000 2024-09-13T14:46:14.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:46:14.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:45:38.454385+0000 2024-09-13T14:46:14.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:46:14.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:46:14.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:46:14.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:46:14.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:46:14.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:46:14.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:46:14.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:46:14.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:45:38.454385+0000 '>' 2024-09-13T14:45:38.454385+0000 2024-09-13T14:46:14.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:46:15.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:46:15.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:46:15.961 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:46:15.961 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:46:15.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:46:15.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:46:15.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:46:16.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:46:14.622559+0000 '>' 2024-09-13T14:45:38.454385+0000 2024-09-13T14:46:16.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:46:16.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:46:18.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:46:18.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:46:18.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:46:18.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-13T14:46:18.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:46:18.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:46:18.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:46:18.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:46:18.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-13T14:46:18.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:46:18.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-13T14:46:18.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:46:18.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:46:18.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:46:18.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:46:18.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:46:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:46:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:46:18.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:46:18.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:46:18.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-13T14:46:18.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:46:18.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:46:18.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-13T14:46:19.357 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:46:19.357 INFO:tasks.workunit.client.0.smithi112.stdout:hinfo_key 2024-09-13T14:46:19.357 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:46:19.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-13T14:46:19.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:46:19.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:46:19.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-13T14:46:19.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:46:19.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:46:19.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:46:19.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:46:19.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:46:19.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:46:19.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-13T14:46:19.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-13T14:46:19.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:46:19.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:46:19.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:46:19.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:46:19.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:46:19.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:46:19.641 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:46:19.641 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:46:19.641 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:46:19.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:46:19.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:46:19.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:46:19.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:46:19.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:46:19.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:46:19.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:46:19.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:46:19.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:46:19.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:46:19.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:46:19.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-13T14:46:19.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-13T14:46:19.642 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.9 2024-09-13T14:46:19.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:46:19.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-13T14:46:19.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-13T14:46:19.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:46:19.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:46:19.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:46:19.683 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:46:19.681+0000 7fa5aa4ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:46:19.686 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:46:19.681+0000 7fa5aa4ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:46:19.687 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:46:19.685+0000 7fa5aa4ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:46:19.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-13T14:46:19.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:46:19.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-13T14:46:19.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:46:19.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:46:19.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:19.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:46:19.984 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:46:19.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:19.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:46:20.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:20.985 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:46:20.981+0000 7fa5aa4ff7c0 -1 Falling back to public interface 2024-09-13T14:46:21.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:21.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:21.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:46:21.362 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:46:21.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:21.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:46:21.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:46:22.697 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:46:22.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:22.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:46:23.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:24.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:24.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:46:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:46:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:46:24.186 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:46:24.181+0000 7fa5aa4ff7c0 -1 osd.9 87 log_to_monitors true 2024-09-13T14:46:24.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:25.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:25.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:25.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:46:25.428 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:46:25.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:25.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:46:25.769 INFO:tasks.workunit.client.0.smithi112.stderr:osd.9 up in weight 1 up_from 90 up_thru 0 down_at 87 last_clean_interval [78,87) [v2:127.0.0.1:6874/932313754,v1:127.0.0.1:6875/932313754] [v2:127.0.0.1:6876/932313754,v1:127.0.0.1:6877/932313754] exists,up 773da5cd-70ef-4087-bab9-35ba28113c89 2024-09-13T14:46:25.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:46:25.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:46:25.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:46:25.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:46:25.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:46:25.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:46:25.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:46:25.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:46:25.770 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:46:25.771 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:46:25.771 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:46:25.771 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:46:25.771 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:46:25.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:46:25.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:46:25.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:46:25.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:46:25.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:46:25.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:46:26.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:46:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:46:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:46:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:46:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:46:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:5 2024-09-13T14:46:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:6 2024-09-13T14:46:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:7 2024-09-13T14:46:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:8 2024-09-13T14:46:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:9' 2024-09-13T14:46:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:46:26.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:26.469 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:46:26.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836574 2024-09-13T14:46:26.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836574 2024-09-13T14:46:26.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574' 2024-09-13T14:46:26.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:26.786 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:46:26.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673045 2024-09-13T14:46:26.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673045 2024-09-13T14:46:26.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673045' 2024-09-13T14:46:26.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:26.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:46:27.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509519 2024-09-13T14:46:27.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509519 2024-09-13T14:46:27.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509519' 2024-09-13T14:46:27.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:27.122 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:46:27.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612624 2024-09-13T14:46:27.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612624 2024-09-13T14:46:27.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509519 3-313532612624' 2024-09-13T14:46:27.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:27.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:46:27.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182466 2024-09-13T14:46:27.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182466 2024-09-13T14:46:27.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509519 3-313532612624 4-107374182466' 2024-09-13T14:46:27.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:27.509 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:46:27.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252871 2024-09-13T14:46:27.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252871 2024-09-13T14:46:27.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509519 3-313532612624 4-107374182466 5-360777252871' 2024-09-13T14:46:27.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:27.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:46:27.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855410 2024-09-13T14:46:27.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855410 2024-09-13T14:46:27.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509519 3-313532612624 4-107374182466 5-360777252871 6-150323855410' 2024-09-13T14:46:27.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:27.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:46:27.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691886 2024-09-13T14:46:27.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691886 2024-09-13T14:46:27.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509519 3-313532612624 4-107374182466 5-360777252871 6-150323855410 7-171798691886' 2024-09-13T14:46:27.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:27.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:46:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528354 2024-09-13T14:46:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528354 2024-09-13T14:46:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509519 3-313532612624 4-107374182466 5-360777252871 6-150323855410 7-171798691886 8-193273528354' 2024-09-13T14:46:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:28.080 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:46:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=386547056642 2024-09-13T14:46:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 386547056642 2024-09-13T14:46:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673045 2-64424509519 3-313532612624 4-107374182466 5-360777252871 6-150323855410 7-171798691886 8-193273528354 9-386547056642' 2024-09-13T14:46:28.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:28.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836574 2024-09-13T14:46:28.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:28.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:46:28.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836574 2024-09-13T14:46:28.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:28.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836574 2024-09-13T14:46:28.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2024-09-13T14:46:28.228 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836574 2024-09-13T14:46:28.228 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:46:28.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836575 -lt 21474836574 2024-09-13T14:46:28.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:28.549 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673045 2024-09-13T14:46:28.549 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:28.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:46:28.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673045 2024-09-13T14:46:28.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:28.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673045 2024-09-13T14:46:28.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-09-13T14:46:28.553 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949673045 2024-09-13T14:46:28.553 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:46:28.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673045 -lt 42949673045 2024-09-13T14:46:28.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:28.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509519 2024-09-13T14:46:28.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:28.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:46:28.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:28.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509519 2024-09-13T14:46:28.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509519 2024-09-13T14:46:28.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509519' 2024-09-13T14:46:28.914 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509519 2024-09-13T14:46:28.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:46:29.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509519 -lt 64424509519 2024-09-13T14:46:29.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:29.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612624 2024-09-13T14:46:29.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:29.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:46:29.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612624 2024-09-13T14:46:29.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:29.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612624 2024-09-13T14:46:29.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612624' 2024-09-13T14:46:29.259 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 313532612624 2024-09-13T14:46:29.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:46:29.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612624 -lt 313532612624 2024-09-13T14:46:29.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:29.609 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182466 2024-09-13T14:46:29.609 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:29.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:46:29.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182466 2024-09-13T14:46:29.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:29.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182466 2024-09-13T14:46:29.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182466' 2024-09-13T14:46:29.613 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.4 seq 107374182466 2024-09-13T14:46:29.613 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:46:30.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182466 -lt 107374182466 2024-09-13T14:46:30.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:30.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-360777252871 2024-09-13T14:46:30.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:30.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:46:30.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-360777252871 2024-09-13T14:46:30.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:30.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252871 2024-09-13T14:46:30.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 360777252871' 2024-09-13T14:46:30.019 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.5 seq 360777252871 2024-09-13T14:46:30.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:46:30.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252872 -lt 360777252871 2024-09-13T14:46:30.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:30.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:30.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855410 2024-09-13T14:46:30.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:46:30.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855410 2024-09-13T14:46:30.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:30.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855410 2024-09-13T14:46:30.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855410' 2024-09-13T14:46:30.355 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.6 seq 150323855410 2024-09-13T14:46:30.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:46:30.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855411 -lt 150323855410 2024-09-13T14:46:30.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:30.745 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691886 2024-09-13T14:46:30.745 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:30.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:46:30.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691886 2024-09-13T14:46:30.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:30.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691886 2024-09-13T14:46:30.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691886' 2024-09-13T14:46:30.749 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.7 seq 171798691886 2024-09-13T14:46:30.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:46:31.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691886 -lt 171798691886 2024-09-13T14:46:31.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:31.102 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528354 2024-09-13T14:46:31.102 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:31.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:46:31.104 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528354 2024-09-13T14:46:31.104 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:31.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528354 2024-09-13T14:46:31.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528354' 2024-09-13T14:46:31.106 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.8 seq 193273528354 2024-09-13T14:46:31.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:46:31.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528354 -lt 193273528354 2024-09-13T14:46:31.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:31.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-386547056642 2024-09-13T14:46:31.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:31.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:46:31.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-386547056642 2024-09-13T14:46:31.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:31.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=386547056642 2024-09-13T14:46:31.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 386547056642' 2024-09-13T14:46:31.459 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.9 seq 386547056642 2024-09-13T14:46:31.459 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:46:31.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 386547056643 -lt 386547056642 2024-09-13T14:46:31.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:46:31.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:46:31.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:46:32.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:46:32.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:46:32.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:46:32.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:46:32.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:46:32.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:46:32.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:46:32.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:46:32.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:46:32.544 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:46:32.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:46:32.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:46:32.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:46:32.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:46:32.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:46:32.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:46:32.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:46:32.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-09-13T14:46:32.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-13T14:46:32.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-13T14:46:32.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-09-13T14:46:32.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-09-13T14:46:32.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-13T14:46:32.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:46:32.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:46:32.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:46:32.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 220353"' 2024-09-13T14:46:32.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 220353' 2024-09-13T14:46:32.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-13T14:46:32.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:46:32.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:46:32.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 220355"' 2024-09-13T14:46:32.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 220355' 2024-09-13T14:46:32.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-13T14:46:32.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:46:32.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 220353 220355' 2024-09-13T14:46:32.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:46:32.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:46:32.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 220353 2024-09-13T14:46:32.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/220356: /' 2024-09-13T14:46:32.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-13T14:46:32.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/220358: /' 2024-09-13T14:46:36.392 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:46:36.392 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:46:36.392 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-13T14:46:36.392 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:46:36.392 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:46:36.392 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:46:36.392 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:46:36.393 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-13T14:46:36.393 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:46:36.393 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-13T14:46:36.393 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:46:36.393 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:46:36.393 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:46:36.393 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:46:36.393 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:46:36.393 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:46:36.393 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:46:36.393 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:46:36.393 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:46:36.393 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-13T14:46:36.393 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:46:36.394 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:46:36.394 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-13T14:46:36.394 INFO:tasks.workunit.client.0.smithi112.stderr:220356: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-13T14:46:36.394 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-13T14:46:36.394 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:46:36.394 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:46:36.394 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-13T14:46:36.394 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:46:36.394 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:46:36.394 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:46:36.394 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:46:36.397 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:46:36.397 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:46:36.397 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-13T14:46:36.397 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-13T14:46:36.397 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:46:36.397 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:46:36.397 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:46:36.397 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:46:36.397 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:46:36.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:46:36.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:46:36.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:46:36.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:46:36.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:46:36.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:46:36.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:46:36.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:46:36.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:46:36.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:46:36.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:46:36.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:46:36.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:46:36.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:46:36.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:46:36.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-13T14:46:36.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-13T14:46:36.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: start osd.5 2024-09-13T14:46:36.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:46:36.466 INFO:tasks.workunit.client.0.smithi112.stderr:2220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:46:36.466 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:46:36.466 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-13T14:46:36.466 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:46:36.466 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-09-13T14:46:36.466 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:46:36.466 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:46:36.466 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-13T14:46:36.466 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:46:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-13T14:46:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:46:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:46:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:46:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:46:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:46:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:46:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-09-13T14:46:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:46:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:46:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-13T14:46:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:46:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:46:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-09-13T14:46:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:220358: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-13T14:46:36.468 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-13T14:46:36.468 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:46:36.468 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:46:36.468 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-13T14:46:36.468 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:46:36.468 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:46:36.468 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:46:36.468 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:46:36.469 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:46:36.470 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:46:36.470 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-13T14:46:36.470 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-13T14:46:36.470 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:46:36.470 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:46:36.470 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:46:36.470 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:46:36.470 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:46:36.470 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:46:36.470 INFO:tasks.workunit.client.0.smithi112.stderr:220358: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:46:36.470 INFO:tasks.workunit.client.0.smithi112.stderr:220358: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:46:36.470 INFO:tasks.workunit.client.0.smithi112.stderr:220358: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:46:36.470 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:46:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:46:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:46:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:46:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:46:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:46:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:46:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:46:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:46:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:46:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:46:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-13T14:46:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-13T14:46:36.472 INFO:tasks.workunit.client.0.smithi112.stderr:220358: start osd.9 2024-09-13T14:46:36.472 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:46:41.114 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-13T14:46:41.114 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-13T14:46:41.114 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:46:41.114 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:46:41.114 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:46:41.114 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 2024-09-13T14:46:36.429+0000 7ff57cd127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:46:41.114 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 2024-09-13T14:46:36.477+0000 7ff57cd127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:46:41.114 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 2024-09-13T14:46:36.477+0000 7ff57cd127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:46:41.115 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-13T14:46:41.115 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:46:41.115 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-13T14:46:41.115 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:46:41.115 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:46:41.115 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:41.115 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:46:41.115 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 0 2024-09-13T14:46:41.115 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:46:41.115 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:41.115 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:41.115 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 2024-09-13T14:46:37.548+0000 7ff57cd127c0 -1 Falling back to public interface 2024-09-13T14:46:41.115 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:41.115 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:46:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 1 2024-09-13T14:46:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:46:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:46:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 2 2024-09-13T14:46:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:46:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 2024-09-13T14:46:40.560+0000 7ff57cd127c0 -1 osd.5 92 log_to_monitors true 2024-09-13T14:46:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:41.116 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:41.117 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:46:41.117 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 3 2024-09-13T14:46:41.117 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:41.117 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:46:41.301 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers20358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-13T14:46:41.301 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-13T14:46:41.301 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:46:41.301 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:46:41.301 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:46:41.301 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 2024-09-13T14:46:36.496+0000 7fb0e93697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:46:41.302 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 2024-09-13T14:46:36.496+0000 7fb0e93697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:46:41.302 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 2024-09-13T14:46:36.500+0000 7fb0e93697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:46:41.302 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-13T14:46:41.302 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:46:41.302 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-13T14:46:41.302 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:46:41.302 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:46:41.302 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:41.302 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:46:41.302 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 0 2024-09-13T14:46:41.302 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:41.302 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:46:41.302 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:41.303 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 2024-09-13T14:46:38.040+0000 7fb0e93697c0 -1 Falling back to public interface 2024-09-13T14:46:41.303 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:41.303 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:41.303 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:46:41.303 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 1 2024-09-13T14:46:41.303 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:41.303 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:46:41.303 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:41.303 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:41.303 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:41.303 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:46:41.303 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 2 2024-09-13T14:46:41.303 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:41.304 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:46:41.304 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:41.304 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:41.304 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:41.304 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:46:41.304 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 3 2024-09-13T14:46:41.304 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:41.304 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:46:41.304 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 2024-09-13T14:46:41.196+0000 7fb0e93697c0 -1 osd.9 92 log_to_monitors true 2024-09-13T14:46:43.398 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:43.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:43.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:43.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:46:43.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 4 2024-09-13T14:46:43.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:43.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:46:43.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: osd.5 up in weight 1 up_from 96 up_thru 96 down_at 93 last_clean_interval [84,92) [v2:127.0.0.1:6842/1234113164,v1:127.0.0.1:6843/1234113164] [v2:127.0.0.1:6844/1234113164,v1:127.0.0.1:6845/1234113164] exists,up ac5b44cf-f8cb-4d05-8d84-3858f28ea531 2024-09-13T14:46:43.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:46:43.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:46:43.398 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:46:43.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:46:43.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:46:43.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:46:43.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:46:43.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:46:43.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:46:43.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:46:43.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:46:43.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:46:43.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:46:43.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:46:43.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:46:43.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:46:43.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:46:43.399 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:46:43.400 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:46:43.400 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:46:43.400 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 1 2024-09-13T14:46:43.400 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 2 2024-09-13T14:46:43.400 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 3 2024-09-13T14:46:43.400 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 4 2024-09-13T14:46:43.400 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 5 2024-09-13T14:46:43.400 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 6 2024-09-13T14:46:43.400 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 7 2024-09-13T14:46:43.400 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 8 2024-09-13T14:46:43.400 INFO:tasks.workunit.client.0.smithi112.stderr:220356: 9' 2024-09-13T14:46:43.400 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:46:43.400 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:43.400 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:46:43.400 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836579 2024-09-13T14:46:43.401 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836579 2024-09-13T14:46:43.401 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579' 2024-09-13T14:46:43.401 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:44.932 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:44.932 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:44.932 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:44.932 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:46:44.932 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 4 2024-09-13T14:46:44.933 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:44.933 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:46:44.933 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:46:44.933 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:46:44.933 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:46:44.933 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:46:44.933 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 5 2024-09-13T14:46:44.933 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:46:44.933 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:46:44.933 INFO:tasks.workunit.client.0.smithi112.stderr:220358: osd.9 up in weight 1 up_from 99 up_thru 0 down_at 93 last_clean_interval [90,92) [v2:127.0.0.1:6874/665706839,v1:127.0.0.1:6875/665706839] [v2:127.0.0.1:6876/665706839,v1:127.0.0.1:6877/665706839] exists,up 773da5cd-70ef-4087-bab9-35ba28113c89 2024-09-13T14:46:44.933 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:46:44.933 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:46:44.933 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:46:44.933 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:46:44.934 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:46:44.934 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:46:44.934 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:46:44.934 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:46:44.934 INFO:tasks.workunit.client.0.smithi112.stderr:220358: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:46:44.934 INFO:tasks.workunit.client.0.smithi112.stderr:220358: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:46:44.934 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:46:44.934 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:46:44.934 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:46:44.934 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:46:44.934 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:46:44.934 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:46:44.934 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:46:44.935 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:46:44.935 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:46:44.935 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:46:44.935 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 1 2024-09-13T14:46:44.935 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 2 2024-09-13T14:46:44.935 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 3 2024-09-13T14:46:44.935 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 4 2024-09-13T14:46:44.935 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 5 2024-09-13T14:46:44.935 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 6 2024-09-13T14:46:44.935 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 7 2024-09-13T14:46:44.935 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 8 2024-09-13T14:46:44.935 INFO:tasks.workunit.client.0.smithi112.stderr:220358: 9' 2024-09-13T14:46:44.935 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:46:45.166 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:46:45.166 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673049 2024-09-13T14:46:45.166 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673049 2024-09-13T14:46:45.166 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673049' 2024-09-13T14:46:45.166 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:45.166 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:46:45.167 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509523 2024-09-13T14:46:45.167 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509523 2024-09-13T14:46:45.167 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509523' 2024-09-13T14:46:45.167 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:45.167 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:46:45.167 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612628 2024-09-13T14:46:45.167 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612628 2024-09-13T14:46:45.167 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509523 3-313532612628' 2024-09-13T14:46:45.167 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:45.167 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:46:45.167 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182470 2024-09-13T14:46:45.167 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182470 2024-09-13T14:46:45.167 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509523 3-313532612628 4-107374182470' 2024-09-13T14:46:45.167 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:45.168 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:46:45.168 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860418 2024-09-13T14:46:45.168 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860418 2024-09-13T14:46:45.168 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509523 3-313532612628 4-107374182470 5-412316860418' 2024-09-13T14:46:45.168 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:45.168 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:46:45.168 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855415 2024-09-13T14:46:45.168 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855415 2024-09-13T14:46:45.168 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509523 3-313532612628 4-107374182470 5-412316860418 6-150323855415' 2024-09-13T14:46:45.168 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:45.168 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:46:45.870 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691890 2024-09-13T14:46:45.870 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691890 2024-09-13T14:46:45.870 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509523 3-313532612628 4-107374182470 5-412316860418 6-150323855415 7-171798691890' 2024-09-13T14:46:45.870 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:45.870 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:46:45.870 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528358 2024-09-13T14:46:45.870 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528358 2024-09-13T14:46:45.870 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509523 3-313532612628 4-107374182470 5-412316860418 6-150323855415 7-171798691890 8-193273528358' 2024-09-13T14:46:45.870 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:45.870 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:46:45.871 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=425201762306 2024-09-13T14:46:45.871 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 425201762306 2024-09-13T14:46:45.871 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509523 3-313532612628 4-107374182470 5-412316860418 6-150323855415 7-171798691890 8-193273528358 9-425201762306' 2024-09-13T14:46:45.871 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:45.871 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836579 2024-09-13T14:46:45.871 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:45.871 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:46:45.871 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836579 2024-09-13T14:46:45.871 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:45.871 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836579 2024-09-13T14:46:45.871 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-09-13T14:46:45.871 INFO:tasks.workunit.client.0.smithi112.stderr:220356: waiting osd.0 seq 21474836579 2024-09-13T14:46:45.871 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:46:45.872 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836579 -lt 21474836579 2024-09-13T14:46:45.872 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:45.872 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673049 2024-09-13T14:46:45.872 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:45.872 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:46:45.872 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673049 2024-09-13T14:46:45.872 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:45.872 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673049 2024-09-13T14:46:45.872 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-09-13T14:46:45.872 INFO:tasks.workunit.client.0.smithi112.stderr:220356: waiting osd.1 seq 42949673049 2024-09-13T14:46:46.096 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_s2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:46.097 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:46:46.097 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836580 2024-09-13T14:46:46.097 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836580 2024-09-13T14:46:46.097 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580' 2024-09-13T14:46:46.097 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:46.097 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:46:46.097 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673051 2024-09-13T14:46:46.097 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673051 2024-09-13T14:46:46.097 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673051' 2024-09-13T14:46:46.097 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:46.097 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:46:46.097 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509524 2024-09-13T14:46:46.098 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509524 2024-09-13T14:46:46.098 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673051 2-64424509524' 2024-09-13T14:46:46.098 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:46.098 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:46:46.098 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612630 2024-09-13T14:46:46.098 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612630 2024-09-13T14:46:46.098 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673051 2-64424509524 3-313532612630' 2024-09-13T14:46:46.098 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:46.098 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:46:46.098 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182471 2024-09-13T14:46:46.098 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182471 2024-09-13T14:46:46.098 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673051 2-64424509524 3-313532612630 4-107374182471' 2024-09-13T14:46:46.098 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:46.099 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:46:46.099 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860419 2024-09-13T14:46:46.099 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860419 2024-09-13T14:46:46.099 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673051 2-64424509524 3-313532612630 4-107374182471 5-412316860419' 2024-09-13T14:46:46.099 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:46.099 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:46:46.099 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855416 2024-09-13T14:46:46.851 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855416 2024-09-13T14:46:46.851 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673051 2-64424509524 3-313532612630 4-107374182471 5-412316860419 6-150323855416' 2024-09-13T14:46:46.851 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:46.851 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:46:46.851 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691892 2024-09-13T14:46:46.851 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691892 2024-09-13T14:46:46.851 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673051 2-64424509524 3-313532612630 4-107374182471 5-412316860419 6-150323855416 7-171798691892' 2024-09-13T14:46:46.851 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:46.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:46:46.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528359 2024-09-13T14:46:46.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528359 2024-09-13T14:46:46.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673051 2-64424509524 3-313532612630 4-107374182471 5-412316860419 6-150323855416 7-171798691892 8-193273528359' 2024-09-13T14:46:46.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:46:46.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:46:46.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=425201762307 2024-09-13T14:46:46.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 425201762307 2024-09-13T14:46:46.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673051 2-64424509524 3-313532612630 4-107374182471 5-412316860419 6-150323855416 7-171798691892 8-193273528359 9-425201762307' 2024-09-13T14:46:46.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:46.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836580 2024-09-13T14:46:46.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:46.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:46:46.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836580 2024-09-13T14:46:46.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:46.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836580 2024-09-13T14:46:46.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-09-13T14:46:46.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: waiting osd.0 seq 21474836580 2024-09-13T14:46:46.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:46:46.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836580 -lt 21474836580 2024-09-13T14:46:46.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:46.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673051 2024-09-13T14:46:46.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:47.310 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpertats: ceph osd last-stat-seq 1 2024-09-13T14:46:47.310 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673050 -lt 42949673049 2024-09-13T14:46:47.310 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:47.310 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509523 2024-09-13T14:46:47.310 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:47.310 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:46:47.310 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509523 2024-09-13T14:46:47.310 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:47.310 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509523 2024-09-13T14:46:47.311 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509523' 2024-09-13T14:46:47.311 INFO:tasks.workunit.client.0.smithi112.stderr:220356: waiting osd.2 seq 64424509523 2024-09-13T14:46:47.311 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:46:47.311 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509524 -lt 64424509523 2024-09-13T14:46:47.311 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:47.311 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612628 2024-09-13T14:46:47.311 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:47.311 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:46:47.311 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612628 2024-09-13T14:46:47.311 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:47.311 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612628 2024-09-13T14:46:47.311 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612628' 2024-09-13T14:46:47.311 INFO:tasks.workunit.client.0.smithi112.stderr:220356: waiting osd.3 seq 313532612628 2024-09-13T14:46:47.311 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:46:47.312 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612630 -lt 313532612628 2024-09-13T14:46:47.312 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:47.312 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182470 2024-09-13T14:46:47.312 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:47.312 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:46:47.312 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182470 2024-09-13T14:46:47.312 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:47.312 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182470 2024-09-13T14:46:47.312 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182470' 2024-09-13T14:46:47.312 INFO:tasks.workunit.client.0.smithi112.stderr:220356: waiting osd.4 seq 107374182470 2024-09-13T14:46:47.312 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:46:47.312 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182471 -lt 107374182470 2024-09-13T14:46:47.312 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:47.312 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-412316860418 2024-09-13T14:46:47.851 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/ces.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:46:47.851 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673051 2024-09-13T14:46:47.851 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:47.851 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673051 2024-09-13T14:46:47.851 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-09-13T14:46:47.851 INFO:tasks.workunit.client.0.smithi112.stderr:220358: waiting osd.1 seq 42949673051 2024-09-13T14:46:47.851 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:46:47.851 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673051 -lt 42949673051 2024-09-13T14:46:47.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:47.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:47.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509524 2024-09-13T14:46:47.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:46:47.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509524 2024-09-13T14:46:47.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:47.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509524 2024-09-13T14:46:47.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-09-13T14:46:47.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: waiting osd.2 seq 64424509524 2024-09-13T14:46:47.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:46:47.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509524 -lt 64424509524 2024-09-13T14:46:47.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:47.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612630 2024-09-13T14:46:47.852 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:47.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:46:47.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612630 2024-09-13T14:46:47.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:47.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612630 2024-09-13T14:46:47.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612630' 2024-09-13T14:46:47.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: waiting osd.3 seq 313532612630 2024-09-13T14:46:47.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:46:47.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612630 -lt 313532612630 2024-09-13T14:46:47.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:47.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182471 2024-09-13T14:46:47.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:47.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:46:47.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182471 2024-09-13T14:46:47.853 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:47.854 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182471 2024-09-13T14:46:48.333 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0phtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:48.333 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:46:48.333 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-412316860418 2024-09-13T14:46:48.333 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:48.333 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860418 2024-09-13T14:46:48.333 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 412316860418' 2024-09-13T14:46:48.333 INFO:tasks.workunit.client.0.smithi112.stderr:220356: waiting osd.5 seq 412316860418 2024-09-13T14:46:48.333 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:46:48.334 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860420 -lt 412316860418 2024-09-13T14:46:48.334 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:48.334 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855415 2024-09-13T14:46:48.334 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:48.334 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:46:48.334 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855415 2024-09-13T14:46:48.334 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:48.334 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855415 2024-09-13T14:46:48.334 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855415' 2024-09-13T14:46:48.334 INFO:tasks.workunit.client.0.smithi112.stderr:220356: waiting osd.6 seq 150323855415 2024-09-13T14:46:48.334 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:46:48.334 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855416 -lt 150323855415 2024-09-13T14:46:48.334 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:48.335 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691890 2024-09-13T14:46:48.335 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:48.335 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:46:48.335 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691890 2024-09-13T14:46:48.335 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:48.335 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691890 2024-09-13T14:46:48.335 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691890' 2024-09-13T14:46:48.335 INFO:tasks.workunit.client.0.smithi112.stderr:220356: waiting osd.7 seq 171798691890 2024-09-13T14:46:48.335 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:46:48.335 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691891 -lt 171798691890 2024-09-13T14:46:48.335 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:48.335 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528358 2024-09-13T14:46:48.335 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:48.335 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:46:48.336 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528358 2024-09-13T14:46:49.156 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: c/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182471' 2024-09-13T14:46:49.157 INFO:tasks.workunit.client.0.smithi112.stderr:220358: waiting osd.4 seq 107374182471 2024-09-13T14:46:49.157 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:46:49.157 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182471 -lt 107374182471 2024-09-13T14:46:49.157 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:49.157 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-412316860419 2024-09-13T14:46:49.157 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:49.157 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:46:49.157 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-412316860419 2024-09-13T14:46:49.157 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:49.157 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860419 2024-09-13T14:46:49.157 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 412316860419' 2024-09-13T14:46:49.158 INFO:tasks.workunit.client.0.smithi112.stderr:220358: waiting osd.5 seq 412316860419 2024-09-13T14:46:49.158 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:46:49.158 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860420 -lt 412316860419 2024-09-13T14:46:49.158 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:49.158 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855416 2024-09-13T14:46:49.158 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:49.158 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:46:49.158 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855416 2024-09-13T14:46:49.158 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:49.158 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855416 2024-09-13T14:46:49.159 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855416' 2024-09-13T14:46:49.159 INFO:tasks.workunit.client.0.smithi112.stderr:220358: waiting osd.6 seq 150323855416 2024-09-13T14:46:49.159 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:46:49.159 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855416 -lt 150323855416 2024-09-13T14:46:49.159 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:49.159 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691892 2024-09-13T14:46:49.159 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:49.159 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:46:49.159 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691892 2024-09-13T14:46:49.159 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:49.160 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691892 2024-09-13T14:46:49.160 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691892' 2024-09-13T14:46:49.160 INFO:tasks.workunit.client.0.smithi112.stderr:220358: waiting osd.7 seq 171798691892 2024-09-13T14:46:49.160 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:46:49.160 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691892 -lt 171798691892 2024-09-13T14:46:50.132 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/ceput -d - -f 2 2024-09-13T14:46:50.132 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528358 2024-09-13T14:46:50.132 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2024-09-13T14:46:50.132 INFO:tasks.workunit.client.0.smithi112.stderr:220356: waiting osd.8 seq 193273528358 2024-09-13T14:46:50.132 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:46:50.132 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528360 -lt 193273528358 2024-09-13T14:46:50.132 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:50.132 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-425201762306 2024-09-13T14:46:50.132 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:50.132 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:46:50.133 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-425201762306 2024-09-13T14:46:50.133 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:50.133 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=425201762306 2024-09-13T14:46:50.133 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 425201762306' 2024-09-13T14:46:50.133 INFO:tasks.workunit.client.0.smithi112.stderr:220356: waiting osd.9 seq 425201762306 2024-09-13T14:46:50.133 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:46:50.133 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 425201762308 -lt 425201762306 2024-09-13T14:46:50.133 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:46:50.133 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:46:50.133 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:46:50.133 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:46:50.133 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:46:50.134 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:46:50.134 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:46:50.134 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:46:50.134 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:46:50.134 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:46:50.134 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:46:50.134 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:46:50.134 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:46:50.134 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:46:50.134 INFO:tasks.workunit.client.0.smithi112.stderr:220356: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:46:50.134 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:46:50.135 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:46:50.135 INFO:tasks.workunit.client.0.smithi112.stderr:220356: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:46:50.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:46:50.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:46:50.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 220355 2024-09-13T14:46:50.542 INFO:tasks.workunit.client.0.smithi112.stderr:htest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:50.542 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528359 2024-09-13T14:46:50.542 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:50.542 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:46:50.542 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528359 2024-09-13T14:46:50.542 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:50.542 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528359 2024-09-13T14:46:50.542 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528359' 2024-09-13T14:46:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:220358: waiting osd.8 seq 193273528359 2024-09-13T14:46:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:46:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528360 -lt 193273528359 2024-09-13T14:46:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:46:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-425201762307 2024-09-13T14:46:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:46:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:46:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-425201762307 2024-09-13T14:46:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:46:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=425201762307 2024-09-13T14:46:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 425201762307' 2024-09-13T14:46:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:220358: waiting osd.9 seq 425201762307 2024-09-13T14:46:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:46:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 425201762308 -lt 425201762307 2024-09-13T14:46:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:46:50.544 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:46:50.544 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:46:50.544 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:46:50.544 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:46:50.544 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:46:50.544 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:46:50.544 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:46:50.544 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:46:50.544 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:46:50.544 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:46:50.544 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:46:50.914 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:46:50.915 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:46:50.915 INFO:tasks.workunit.client.0.smithi112.stderr:220358: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:46:50.915 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:46:50.915 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:46:50.915 INFO:tasks.workunit.client.0.smithi112.stderr:220358: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:46:50.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:46:50.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:46:50.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:46:50.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:46:50.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-13T14:46:50.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:46:50.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-13T14:46:50.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:46:50.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:46:50.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:46:50.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:46:51.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-13T14:46:51.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-13T14:46:51.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:46:51.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:46:51.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:46:51.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:46:51.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:46:51.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:46:51.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:46:14.622559+0000 2024-09-13T14:46:51.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:46:51.912 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:46:51.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:46:14.622559+0000 2024-09-13T14:46:51.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:46:51.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:46:14.622559+0000 2024-09-13T14:46:51.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:46:51.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:46:51.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:46:51.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:46:51.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:46:51.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:46:51.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:46:51.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:46:52.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:46:14.622559+0000 '>' 2024-09-13T14:46:14.622559+0000 2024-09-13T14:46:52.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:46:53.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:46:53.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:46:53.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:46:53.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:46:53.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:46:53.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:46:53.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:46:53.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:46:14.622559+0000 '>' 2024-09-13T14:46:14.622559+0000 2024-09-13T14:46:53.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:46:54.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:46:54.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:46:54.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:46:54.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:46:54.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:46:54.591 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:46:54.591 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:46:54.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:46:14.622559+0000 '>' 2024-09-13T14:46:14.622559+0000 2024-09-13T14:46:54.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:46:55.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:46:55.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:46:55.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:46:55.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:46:55.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:46:55.931 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:46:55.931 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:46:56.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:46:52.809632+0000 '>' 2024-09-13T14:46:14.622559+0000 2024-09-13T14:46:56.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:46:56.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:46:58.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-13T14:46:58.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:46:58.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:46:58.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:46:58.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 223827"' 2024-09-13T14:46:58.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 223827' 2024-09-13T14:46:58.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:46:58.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:46:58.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:46:58.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 223829"' 2024-09-13T14:46:58.270 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 223829' 2024-09-13T14:46:58.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-13T14:46:58.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 223827 223829' 2024-09-13T14:46:58.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:46:58.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:46:58.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 223827 2024-09-13T14:46:58.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:46:58.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/223830: /' 2024-09-13T14:46:58.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:46:58.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/223832: /' 2024-09-13T14:46:59.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:46:59.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:46:59.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-13T14:46:59.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:46:59.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:46:59.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:46:59.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:46:59.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-13T14:46:59.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:46:59.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-13T14:46:59.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:46:59.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:46:59.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:46:59.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:46:59.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:46:59.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:46:59.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:46:59.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:46:59.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:46:59.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-13T14:46:59.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:46:59.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:46:59.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-13T14:46:59.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: _ 2024-09-13T14:46:59.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: hinfo_key 2024-09-13T14:46:59.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: snapset 2024-09-13T14:46:59.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-13T14:46:59.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:46:59.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:46:59.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-13T14:46:59.460 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:46:59.460 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:46:59.460 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:46:59.460 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:46:59.462 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:46:59.462 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:46:59.462 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-13T14:46:59.463 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-13T14:46:59.463 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:46:59.463 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:46:59.463 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:46:59.463 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:46:59.463 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:46:59.463 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:46:59.463 INFO:tasks.workunit.client.0.smithi112.stderr:223830: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:46:59.463 INFO:tasks.workunit.client.0.smithi112.stderr:223830: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:46:59.463 INFO:tasks.workunit.client.0.smithi112.stderr:223830: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:46:59.463 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:46:59.463 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:46:59.464 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:46:59.464 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:46:59.464 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:46:59.464 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:46:59.464 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:46:59.464 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:46:59.464 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:46:59.464 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:46:59.464 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:46:59.464 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-13T14:46:59.464 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-13T14:46:59.464 INFO:tasks.workunit.client.0.smithi112.stderr:223830: start osd.5 2024-09-13T14:46:59.499 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:46:59.499 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:46:59.499 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-13T14:46:59.499 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:46:59.499 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:46:59.499 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:46:59.499 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:46:59.499 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-13T14:46:59.500 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:46:59.500 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-13T14:46:59.500 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:46:59.500 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:46:59.500 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:46:59.500 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:46:59.500 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:46:59.500 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:46:59.500 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:46:59.500 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:46:59.500 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:46:59.500 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-13T14:46:59.500 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:46:59.500 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:46:59.501 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-13T14:46:59.501 INFO:tasks.workunit.client.0.smithi112.stderr:223832: _ 2024-09-13T14:46:59.501 INFO:tasks.workunit.client.0.smithi112.stderr:223832: hinfo_key 2024-09-13T14:46:59.501 INFO:tasks.workunit.client.0.smithi112.stderr:223832: snapset 2024-09-13T14:46:59.501 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-13T14:46:59.501 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:46:59.501 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:46:59.501 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-13T14:46:59.501 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:46:59.501 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:46:59.501 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:46:59.501 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:46:59.503 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:46:59.503 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:46:59.503 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-13T14:46:59.503 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-13T14:46:59.503 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:46:59.503 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:46:59.503 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:46:59.503 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:46:59.503 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:46:59.503 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:46:59.503 INFO:tasks.workunit.client.0.smithi112.stderr:223832: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:46:59.503 INFO:tasks.workunit.client.0.smithi112.stderr:223832: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:46:59.504 INFO:tasks.workunit.client.0.smithi112.stderr:223832: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:46:59.504 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:46:59.504 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:46:59.504 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:46:59.504 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:46:59.504 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:46:59.504 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:46:59.504 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:46:59.504 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:46:59.504 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:46:59.504 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:46:59.504 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:46:59.504 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-13T14:46:59.505 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-13T14:46:59.505 INFO:tasks.workunit.client.0.smithi112.stderr:223832: start osd.9 2024-09-13T14:47:04.456 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:47:04.457 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-13T14:47:04.457 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-13T14:47:04.457 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:47:04.457 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:47:04.457 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:47:04.457 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 2024-09-13T14:46:59.540+0000 7f14163257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:04.457 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 2024-09-13T14:46:59.540+0000 7f14163257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:04.457 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 2024-09-13T14:46:59.544+0000 7f14163257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:04.457 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-13T14:47:04.458 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:47:04.458 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-13T14:47:04.458 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:47:04.458 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:47:04.458 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:04.458 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:47:04.458 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 0 2024-09-13T14:47:04.458 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:04.458 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:47:04.458 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:04.458 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 2024-09-13T14:47:00.824+0000 7f14163257c0 -1 Falling back to public interface 2024-09-13T14:47:04.458 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:04.459 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:04.459 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:47:04.459 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 1 2024-09-13T14:47:04.459 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:04.459 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:47:04.459 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:04.459 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:04.459 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:04.459 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:47:04.459 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 2 2024-09-13T14:47:04.459 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:04.459 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:47:04.459 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:04.460 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 2024-09-13T14:47:03.952+0000 7f14163257c0 -1 osd.9 101 log_to_monitors true 2024-09-13T14:47:04.460 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:04.460 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:04.460 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:47:04.460 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 3 2024-09-13T14:47:04.460 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:04.460 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:47:05.456 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cvery_ops 2024-09-13T14:47:05.456 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-13T14:47:05.456 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-13T14:47:05.456 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:47:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:47:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:47:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 2024-09-13T14:46:59.496+0000 7fdd2ecad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 2024-09-13T14:46:59.500+0000 7fdd2ecad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 2024-09-13T14:46:59.500+0000 7fdd2ecad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-13T14:47:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:47:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-13T14:47:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:47:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:47:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:47:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 0 2024-09-13T14:47:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 2024-09-13T14:47:01.296+0000 7fdd2ecad7c0 -1 Falling back to public interface 2024-09-13T14:47:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:47:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 1 2024-09-13T14:47:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:47:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 2 2024-09-13T14:47:05.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:05.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:05.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:05.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:05.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:05.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:47:05.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 3 2024-09-13T14:47:05.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:05.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:05.459 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:06.764 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.clieneph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:06.764 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:06.764 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:06.764 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:47:06.764 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 4 2024-09-13T14:47:06.765 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:06.765 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:47:06.765 INFO:tasks.workunit.client.0.smithi112.stderr:223832: osd.9 up in weight 1 up_from 106 up_thru 0 down_at 102 last_clean_interval [99,101) [v2:127.0.0.1:6842/2264794401,v1:127.0.0.1:6843/2264794401] [v2:127.0.0.1:6844/2264794401,v1:127.0.0.1:6845/2264794401] exists,up 773da5cd-70ef-4087-bab9-35ba28113c89 2024-09-13T14:47:06.765 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:47:06.765 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:47:06.765 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:47:06.765 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:47:06.765 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:47:06.765 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:47:06.765 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:47:06.765 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:47:06.765 INFO:tasks.workunit.client.0.smithi112.stderr:223832: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:47:06.765 INFO:tasks.workunit.client.0.smithi112.stderr:223832: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:47:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:47:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:47:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:47:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:47:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:47:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:47:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:47:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:47:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:47:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:47:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 1 2024-09-13T14:47:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 2 2024-09-13T14:47:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 3 2024-09-13T14:47:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 4 2024-09-13T14:47:06.767 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 5 2024-09-13T14:47:06.767 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 6 2024-09-13T14:47:06.767 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 7 2024-09-13T14:47:06.767 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 8 2024-09-13T14:47:06.767 INFO:tasks.workunit.client.0.smithi112.stderr:223832: 9' 2024-09-13T14:47:06.767 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:47:06.767 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:06.767 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:47:06.767 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836585 2024-09-13T14:47:06.767 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836585 2024-09-13T14:47:06.767 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585' 2024-09-13T14:47:08.248 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $t.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:08.249 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:08.249 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:47:08.249 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 4 2024-09-13T14:47:08.249 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:08.249 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:08.249 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:08.249 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 2024-09-13T14:47:06.272+0000 7fdd2ecad7c0 -1 osd.5 101 log_to_monitors true 2024-09-13T14:47:08.249 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:08.249 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:08.249 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:47:08.249 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 5 2024-09-13T14:47:08.249 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:08.249 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:08.250 INFO:tasks.workunit.client.0.smithi112.stderr:223830: osd.5 up in weight 1 up_from 110 up_thru 96 down_at 102 last_clean_interval [96,101) [v2:127.0.0.1:6874/2412189076,v1:127.0.0.1:6875/2412189076] [v2:127.0.0.1:6876/2412189076,v1:127.0.0.1:6877/2412189076] exists,up ac5b44cf-f8cb-4d05-8d84-3858f28ea531 2024-09-13T14:47:08.250 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:47:08.250 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:47:08.250 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:47:08.250 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:47:08.250 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:47:08.250 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:47:08.250 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:47:08.250 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:47:08.250 INFO:tasks.workunit.client.0.smithi112.stderr:223830: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:47:08.250 INFO:tasks.workunit.client.0.smithi112.stderr:223830: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:47:08.250 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:47:08.250 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:47:08.250 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:47:08.251 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:47:08.251 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:47:08.251 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:47:08.251 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:47:08.251 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:47:08.251 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:47:08.251 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:47:08.251 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 1 2024-09-13T14:47:08.251 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 2 2024-09-13T14:47:08.251 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 3 2024-09-13T14:47:08.251 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 4 2024-09-13T14:47:08.251 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 5 2024-09-13T14:47:08.251 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 6 2024-09-13T14:47:08.252 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 7 2024-09-13T14:47:08.252 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 8 2024-09-13T14:47:08.252 INFO:tasks.workunit.client.0.smithi112.stderr:223830: 9' 2024-09-13T14:47:08.252 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:47:08.460 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standaloids 2024-09-13T14:47:08.461 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:47:08.461 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673056 2024-09-13T14:47:08.461 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673056 2024-09-13T14:47:08.461 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673056' 2024-09-13T14:47:08.461 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:08.461 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:47:08.461 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509530 2024-09-13T14:47:08.461 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509530 2024-09-13T14:47:08.461 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673056 2-64424509530' 2024-09-13T14:47:08.461 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:08.461 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:47:08.462 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612635 2024-09-13T14:47:08.462 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612635 2024-09-13T14:47:08.462 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673056 2-64424509530 3-313532612635' 2024-09-13T14:47:08.462 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:08.462 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:47:08.462 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182477 2024-09-13T14:47:08.462 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182477 2024-09-13T14:47:08.462 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673056 2-64424509530 3-313532612635 4-107374182477' 2024-09-13T14:47:08.462 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:08.462 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:47:08.462 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=472446402562 2024-09-13T14:47:08.462 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 472446402562 2024-09-13T14:47:08.462 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673056 2-64424509530 3-313532612635 4-107374182477 5-472446402562' 2024-09-13T14:47:08.463 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:08.463 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:47:08.463 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855421 2024-09-13T14:47:08.463 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855421 2024-09-13T14:47:08.463 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673056 2-64424509530 3-313532612635 4-107374182477 5-472446402562 6-150323855421' 2024-09-13T14:47:08.463 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:08.463 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:47:09.563 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691897 2024-09-13T14:47:09.563 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691897 2024-09-13T14:47:09.564 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673056 2-64424509530 3-313532612635 4-107374182477 5-472446402562 6-150323855421 7-171798691897' 2024-09-13T14:47:09.564 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:09.564 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:47:09.564 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528365 2024-09-13T14:47:09.564 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528365 2024-09-13T14:47:09.564 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673056 2-64424509530 3-313532612635 4-107374182477 5-472446402562 6-150323855421 7-171798691897 8-193273528365' 2024-09-13T14:47:09.564 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:09.564 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:47:09.564 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533378 2024-09-13T14:47:09.564 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533378 2024-09-13T14:47:09.564 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673056 2-64424509530 3-313532612635 4-107374182477 5-472446402562 6-150323855421 7-171798691897 8-193273528365 9-455266533378' 2024-09-13T14:47:09.564 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:09.564 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836585 2024-09-13T14:47:09.565 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:09.565 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:47:09.565 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836585 2024-09-13T14:47:09.565 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:09.565 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836585 2024-09-13T14:47:09.565 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-09-13T14:47:09.565 INFO:tasks.workunit.client.0.smithi112.stderr:223832: waiting osd.0 seq 21474836585 2024-09-13T14:47:09.565 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:47:09.565 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836586 -lt 21474836585 2024-09-13T14:47:09.565 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:09.565 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673056 2024-09-13T14:47:09.565 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:09.565 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:47:09.566 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673056 2024-09-13T14:47:09.566 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:09.566 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673056 2024-09-13T14:47:09.566 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-09-13T14:47:09.566 INFO:tasks.workunit.client.0.smithi112.stderr:223832: waiting osd.1 seq 42949673056 2024-09-13T14:47:09.728 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:222ne/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:09.728 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:47:09.728 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836587 2024-09-13T14:47:09.728 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836587 2024-09-13T14:47:09.728 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587' 2024-09-13T14:47:09.728 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:09.729 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:47:09.729 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673057 2024-09-13T14:47:09.729 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673057 2024-09-13T14:47:09.729 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673057' 2024-09-13T14:47:09.729 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:09.729 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:47:09.729 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509531 2024-09-13T14:47:09.729 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509531 2024-09-13T14:47:09.729 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509531' 2024-09-13T14:47:09.729 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:09.729 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:47:09.729 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612636 2024-09-13T14:47:09.729 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612636 2024-09-13T14:47:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509531 3-313532612636' 2024-09-13T14:47:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:47:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182478 2024-09-13T14:47:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182478 2024-09-13T14:47:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509531 3-313532612636 4-107374182478' 2024-09-13T14:47:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:47:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=472446402563 2024-09-13T14:47:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 472446402563 2024-09-13T14:47:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509531 3-313532612636 4-107374182478 5-472446402563' 2024-09-13T14:47:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:09.730 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:47:10.585 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855423 2024-09-13T14:47:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855423 2024-09-13T14:47:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509531 3-313532612636 4-107374182478 5-472446402563 6-150323855423' 2024-09-13T14:47:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:47:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691898 2024-09-13T14:47:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691898 2024-09-13T14:47:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509531 3-313532612636 4-107374182478 5-472446402563 6-150323855423 7-171798691898' 2024-09-13T14:47:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:47:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528366 2024-09-13T14:47:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528366 2024-09-13T14:47:10.586 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509531 3-313532612636 4-107374182478 5-472446402563 6-150323855423 7-171798691898 8-193273528366' 2024-09-13T14:47:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:47:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533380 2024-09-13T14:47:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533380 2024-09-13T14:47:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673057 2-64424509531 3-313532612636 4-107374182478 5-472446402563 6-150323855423 7-171798691898 8-193273528366 9-455266533380' 2024-09-13T14:47:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836587 2024-09-13T14:47:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:47:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836587 2024-09-13T14:47:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836587 2024-09-13T14:47:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-09-13T14:47:10.587 INFO:tasks.workunit.client.0.smithi112.stderr:223830: waiting osd.0 seq 21474836587 2024-09-13T14:47:10.588 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:47:10.588 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836587 -lt 21474836587 2024-09-13T14:47:10.588 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:10.588 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673057 2024-09-13T14:47:10.588 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:10.920 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/sta2: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:47:10.920 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673056 -lt 42949673056 2024-09-13T14:47:10.921 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:10.921 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509530 2024-09-13T14:47:10.921 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:10.921 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:47:10.921 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509530 2024-09-13T14:47:10.921 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:10.921 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509530 2024-09-13T14:47:10.921 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-09-13T14:47:10.921 INFO:tasks.workunit.client.0.smithi112.stderr:223832: waiting osd.2 seq 64424509530 2024-09-13T14:47:10.921 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:47:10.921 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509530 -lt 64424509530 2024-09-13T14:47:10.921 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:10.921 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612635 2024-09-13T14:47:10.922 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:10.922 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:47:10.922 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612635 2024-09-13T14:47:10.922 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:10.922 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612635 2024-09-13T14:47:10.922 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612635' 2024-09-13T14:47:10.922 INFO:tasks.workunit.client.0.smithi112.stderr:223832: waiting osd.3 seq 313532612635 2024-09-13T14:47:10.922 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:47:10.922 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612637 -lt 313532612635 2024-09-13T14:47:10.922 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:10.922 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182477 2024-09-13T14:47:10.922 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:10.922 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:47:10.923 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182477 2024-09-13T14:47:10.923 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:10.923 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182477 2024-09-13T14:47:10.923 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182477' 2024-09-13T14:47:10.923 INFO:tasks.workunit.client.0.smithi112.stderr:223832: waiting osd.4 seq 107374182477 2024-09-13T14:47:10.923 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:47:10.923 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182478 -lt 107374182477 2024-09-13T14:47:10.923 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:10.923 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-472446402562 2024-09-13T14:47:11.621 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //hndalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:47:11.621 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673057 2024-09-13T14:47:11.621 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:11.621 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673057 2024-09-13T14:47:11.621 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-09-13T14:47:11.621 INFO:tasks.workunit.client.0.smithi112.stderr:223830: waiting osd.1 seq 42949673057 2024-09-13T14:47:11.621 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:47:11.621 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673058 -lt 42949673057 2024-09-13T14:47:11.621 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509531 2024-09-13T14:47:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:47:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509531 2024-09-13T14:47:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509531 2024-09-13T14:47:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2024-09-13T14:47:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:223830: waiting osd.2 seq 64424509531 2024-09-13T14:47:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:47:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509531 -lt 64424509531 2024-09-13T14:47:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612636 2024-09-13T14:47:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:11.622 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:47:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612636 2024-09-13T14:47:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612636 2024-09-13T14:47:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612636' 2024-09-13T14:47:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:223830: waiting osd.3 seq 313532612636 2024-09-13T14:47:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:47:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612637 -lt 313532612636 2024-09-13T14:47:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182478 2024-09-13T14:47:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:47:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182478 2024-09-13T14:47:11.623 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:11.624 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182478 2024-09-13T14:47:11.966 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:11.966 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:47:11.966 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-472446402562 2024-09-13T14:47:11.966 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:11.966 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=472446402562 2024-09-13T14:47:11.966 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 472446402562' 2024-09-13T14:47:11.966 INFO:tasks.workunit.client.0.smithi112.stderr:223832: waiting osd.5 seq 472446402562 2024-09-13T14:47:11.966 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:47:11.967 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 472446402563 -lt 472446402562 2024-09-13T14:47:11.967 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:11.967 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855421 2024-09-13T14:47:11.967 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:11.967 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:47:11.967 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855421 2024-09-13T14:47:11.967 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:11.967 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855421 2024-09-13T14:47:11.967 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855421' 2024-09-13T14:47:11.967 INFO:tasks.workunit.client.0.smithi112.stderr:223832: waiting osd.6 seq 150323855421 2024-09-13T14:47:11.968 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:47:11.968 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855423 -lt 150323855421 2024-09-13T14:47:11.968 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:11.968 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:11.968 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691897 2024-09-13T14:47:11.968 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:47:11.968 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691897 2024-09-13T14:47:11.968 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:11.968 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691897 2024-09-13T14:47:11.968 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691897' 2024-09-13T14:47:11.968 INFO:tasks.workunit.client.0.smithi112.stderr:223832: waiting osd.7 seq 171798691897 2024-09-13T14:47:11.969 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:47:11.969 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691898 -lt 171798691897 2024-09-13T14:47:11.969 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:11.969 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528365 2024-09-13T14:47:11.969 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:11.969 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:47:11.969 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528365 2024-09-13T14:47:13.000 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flushtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182478' 2024-09-13T14:47:13.000 INFO:tasks.workunit.client.0.smithi112.stderr:223830: waiting osd.4 seq 107374182478 2024-09-13T14:47:13.000 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:47:13.000 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182478 -lt 107374182478 2024-09-13T14:47:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-472446402563 2024-09-13T14:47:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:47:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-472446402563 2024-09-13T14:47:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=472446402563 2024-09-13T14:47:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 472446402563' 2024-09-13T14:47:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:223830: waiting osd.5 seq 472446402563 2024-09-13T14:47:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:47:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 472446402563 -lt 472446402563 2024-09-13T14:47:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855423 2024-09-13T14:47:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:47:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855423 2024-09-13T14:47:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855423 2024-09-13T14:47:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855423' 2024-09-13T14:47:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:223830: waiting osd.6 seq 150323855423 2024-09-13T14:47:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:47:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855423 -lt 150323855423 2024-09-13T14:47:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691898 2024-09-13T14:47:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:47:13.003 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691898 2024-09-13T14:47:13.003 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:13.003 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691898 2024-09-13T14:47:13.003 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691898' 2024-09-13T14:47:13.003 INFO:tasks.workunit.client.0.smithi112.stderr:223830: waiting osd.7 seq 171798691898 2024-09-13T14:47:13.003 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:47:13.003 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691899 -lt 171798691898 2024-09-13T14:47:13.748 INFO:tasks.workunit.client.0.smithi112.stderr:22383_pg_stats: cut -d - -f 2 2024-09-13T14:47:13.748 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528365 2024-09-13T14:47:13.748 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528365' 2024-09-13T14:47:13.748 INFO:tasks.workunit.client.0.smithi112.stderr:223832: waiting osd.8 seq 193273528365 2024-09-13T14:47:13.748 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:47:13.748 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528366 -lt 193273528365 2024-09-13T14:47:13.748 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:13.748 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-455266533378 2024-09-13T14:47:13.749 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:13.749 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:47:13.749 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-455266533378 2024-09-13T14:47:13.749 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:13.749 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533378 2024-09-13T14:47:13.749 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 455266533378' 2024-09-13T14:47:13.749 INFO:tasks.workunit.client.0.smithi112.stderr:223832: waiting osd.9 seq 455266533378 2024-09-13T14:47:13.749 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:47:13.749 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533380 -lt 455266533378 2024-09-13T14:47:13.749 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:47:13.749 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:47:13.749 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:47:13.750 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:47:13.750 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:47:13.750 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:47:13.750 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:47:13.750 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:47:13.750 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:47:13.750 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:47:13.750 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:47:13.750 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:47:13.750 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:47:13.750 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:47:13.750 INFO:tasks.workunit.client.0.smithi112.stderr:223832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:47:13.750 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:47:13.750 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:47:13.751 INFO:tasks.workunit.client.0.smithi112.stderr:223832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:47:13.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:47:14.426 INFO:tasks.workunit.client.0.smithi112.stderr:0: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:14.426 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528366 2024-09-13T14:47:14.426 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:14.427 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:47:14.427 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528366 2024-09-13T14:47:14.427 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:14.427 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528366 2024-09-13T14:47:14.427 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528366' 2024-09-13T14:47:14.427 INFO:tasks.workunit.client.0.smithi112.stderr:223830: waiting osd.8 seq 193273528366 2024-09-13T14:47:14.427 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:47:14.427 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528367 -lt 193273528366 2024-09-13T14:47:14.427 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:14.427 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-455266533380 2024-09-13T14:47:14.427 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:14.427 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:47:14.428 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-455266533380 2024-09-13T14:47:14.428 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:14.428 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533380 2024-09-13T14:47:14.428 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 455266533380' 2024-09-13T14:47:14.428 INFO:tasks.workunit.client.0.smithi112.stderr:223830: waiting osd.9 seq 455266533380 2024-09-13T14:47:14.428 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:47:14.428 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533380 -lt 455266533380 2024-09-13T14:47:14.428 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:47:14.428 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:47:14.428 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:47:14.428 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:47:14.428 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:47:14.429 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:47:14.429 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:47:14.429 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:47:14.429 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:47:14.429 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:47:14.429 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:47:14.429 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:47:14.797 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:47:14.797 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:47:14.797 INFO:tasks.workunit.client.0.smithi112.stderr:223830: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:47:14.797 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:47:14.798 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:47:14.798 INFO:tasks.workunit.client.0.smithi112.stderr:223830: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:47:14.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:47:14.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:47:14.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 223829 2024-09-13T14:47:14.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:47:14.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:47:14.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:47:14.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-13T14:47:14.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:47:14.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:47:14.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:47:14.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-09-13T14:47:14.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-13T14:47:14.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-13T14:47:14.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-13T14:47:14.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-09-13T14:47:14.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-13T14:47:14.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:47:14.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:47:14.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:47:14.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 226786"' 2024-09-13T14:47:14.836 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 226786' 2024-09-13T14:47:14.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:47:14.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:47:14.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:47:14.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:47:14.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 226788"' 2024-09-13T14:47:14.837 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 226788' 2024-09-13T14:47:14.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/226789: /' 2024-09-13T14:47:14.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-13T14:47:14.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 226786 226788' 2024-09-13T14:47:14.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:47:14.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:47:14.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 226786 2024-09-13T14:47:14.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:47:14.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/226792: /' 2024-09-13T14:47:18.485 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:47:18.485 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:47:18.485 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:47:18.486 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:47:18.486 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:47:18.486 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:47:18.486 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:47:18.486 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:47:18.486 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:47:18.486 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:47:18.486 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:47:18.486 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:47:18.486 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:47:18.486 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:47:18.486 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:47:18.487 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:47:18.487 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:47:18.487 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:47:18.487 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:47:18.487 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:47:18.487 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:47:18.487 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:47:18.487 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-13T14:47:18.487 INFO:tasks.workunit.client.0.smithi112.stderr:226789: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-13T14:47:18.487 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:47:18.487 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:47:18.487 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:47:18.487 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:47:18.488 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:47:18.488 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:47:18.488 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:47:18.488 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:47:18.492 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:47:18.492 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:47:18.492 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:47:18.492 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:47:18.492 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:47:18.492 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:47:18.492 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:47:18.492 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:47:18.492 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:47:18.492 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:47:18.492 INFO:tasks.workunit.client.0.smithi112.stderr:226789: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:47:18.493 INFO:tasks.workunit.client.0.smithi112.stderr:226789: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:47:18.493 INFO:tasks.workunit.client.0.smithi112.stderr:226789: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:47:18.493 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:47:18.493 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:47:18.493 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:47:18.493 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:47:18.493 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:47:18.493 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:47:18.493 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:47:18.493 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:47:18.493 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:47:18.493 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:47:18.493 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:47:18.494 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:47:18.494 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:47:18.494 INFO:tasks.workunit.client.0.smithi112.stderr:226789: start osd.3 2024-09-13T14:47:18.494 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:47:18.603 INFO:tasks.workunit.client.0.smithi112.stderr:2226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:47:18.603 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:47:18.603 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-13T14:47:18.603 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:47:18.603 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:47:18.604 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:47:18.604 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:47:18.604 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-13T14:47:18.604 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:47:18.604 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-13T14:47:18.604 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:47:18.604 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:47:18.604 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:47:18.604 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:47:18.604 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:47:18.605 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:47:18.605 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:47:18.605 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:47:18.605 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:47:18.605 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-13T14:47:18.605 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:47:18.605 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:47:18.605 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-13T14:47:18.605 INFO:tasks.workunit.client.0.smithi112.stderr:226792: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-13T14:47:18.605 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-13T14:47:18.605 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:47:18.605 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:47:18.606 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-13T14:47:18.606 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:47:18.606 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:47:18.606 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:47:18.606 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:47:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:47:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:47:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-13T14:47:18.608 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-13T14:47:18.608 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:47:18.608 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:47:18.608 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:47:18.608 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:47:18.608 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:47:18.608 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:47:18.608 INFO:tasks.workunit.client.0.smithi112.stderr:226792: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:47:18.608 INFO:tasks.workunit.client.0.smithi112.stderr:226792: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:47:18.608 INFO:tasks.workunit.client.0.smithi112.stderr:226792: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:47:18.608 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:47:18.608 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:47:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:47:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:47:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:47:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:47:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:47:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:47:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:47:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:47:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:47:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-13T14:47:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-13T14:47:18.610 INFO:tasks.workunit.client.0.smithi112.stderr:226792: start osd.5 2024-09-13T14:47:18.610 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:47:23.588 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:47:23.588 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:47:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:47:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:47:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:47:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 2024-09-13T14:47:18.523+0000 7f0f6493f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 2024-09-13T14:47:18.527+0000 7f0f6493f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 2024-09-13T14:47:18.527+0000 7f0f6493f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:47:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:47:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:47:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:47:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:47:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:23.589 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:47:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 0 2024-09-13T14:47:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:47:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:47:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 1 2024-09-13T14:47:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:47:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 2024-09-13T14:47:20.307+0000 7f0f6493f7c0 -1 Falling back to public interface 2024-09-13T14:47:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:23.591 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:47:23.591 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 2 2024-09-13T14:47:23.591 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:23.591 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:47:23.591 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:23.591 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:23.591 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:23.591 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:47:23.591 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 3 2024-09-13T14:47:23.591 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:23.591 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:47:23.591 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:23.741 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 2024-09-13T14:47:23.587+0000 7f0f6493f7c0 -26792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-13T14:47:23.741 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-13T14:47:23.741 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:47:23.741 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:47:23.741 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:47:23.741 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 2024-09-13T14:47:18.647+0000 7feffbad17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:23.741 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 2024-09-13T14:47:18.659+0000 7feffbad17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:23.741 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 2024-09-13T14:47:18.663+0000 7feffbad17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:23.741 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-13T14:47:23.742 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:47:23.742 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-13T14:47:23.742 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:47:23.742 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:47:23.742 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:23.742 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:47:23.742 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 0 2024-09-13T14:47:23.742 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:23.742 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:23.742 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:23.742 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 2024-09-13T14:47:20.231+0000 7feffbad17c0 -1 Falling back to public interface 2024-09-13T14:47:23.742 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:23.742 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:23.743 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:47:23.743 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 1 2024-09-13T14:47:23.743 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:23.743 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:23.743 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:23.743 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:23.743 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:23.743 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:47:23.743 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 2 2024-09-13T14:47:23.743 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:23.743 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:23.743 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:23.743 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:23.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:23.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:47:23.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 3 2024-09-13T14:47:23.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:23.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:23.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:25.661 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 2024-09-13T14:47:23.739+0000 7feffbad17c0 -1 osd.5 112 log_to_monitors true 2024-09-13T14:47:25.661 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:25.661 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:25.661 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:47:25.661 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 4 2024-09-13T14:47:25.661 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:25.661 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:25.661 INFO:tasks.workunit.client.0.smithi112.stderr:226792: osd.5 up in weight 1 up_from 118 up_thru 110 down_at 113 last_clean_interval [110,112) [v2:127.0.0.1:6826/2362607248,v1:127.0.0.1:6827/2362607248] [v2:127.0.0.1:6828/2362607248,v1:127.0.0.1:6829/2362607248] exists,up ac5b44cf-f8cb-4d05-8d84-3858f28ea531 2024-09-13T14:47:25.662 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:47:25.662 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:47:25.662 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:47:25.662 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:47:25.662 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:47:25.662 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:47:25.662 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:47:25.662 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:47:25.662 INFO:tasks.workunit.client.0.smithi112.stderr:226792: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:47:25.662 INFO:tasks.workunit.client.0.smithi112.stderr:226792: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:47:25.662 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:47:25.662 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:47:25.662 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:47:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:47:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:47:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:47:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:47:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:47:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:47:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:47:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 1 2024-09-13T14:47:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 2 2024-09-13T14:47:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 3 2024-09-13T14:47:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 4 2024-09-13T14:47:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 5 2024-09-13T14:47:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 6 2024-09-13T14:47:25.663 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 7 2024-09-13T14:47:25.664 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 8 2024-09-13T14:47:25.664 INFO:tasks.workunit.client.0.smithi112.stderr:226792: 9' 2024-09-13T14:47:25.664 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:47:25.664 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:25.664 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:47:25.664 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836591 2024-09-13T14:47:25.664 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836591 2024-09-13T14:47:25.664 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591' 2024-09-13T14:47:25.664 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:25.664 INFO:tasks.workunit.client.0.smithi112.stderr:2261 osd.3 112 log_to_monitors true 2024-09-13T14:47:25.664 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:25.664 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:25.664 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:47:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 4 2024-09-13T14:47:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:47:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:226789: osd.3 up in weight 1 up_from 118 up_thru 110 down_at 113 last_clean_interval [73,112) [v2:127.0.0.1:6874/2076203043,v1:127.0.0.1:6875/2076203043] [v2:127.0.0.1:6876/2076203043,v1:127.0.0.1:6877/2076203043] exists,up 240beeea-9a32-48d8-9b95-819a8b7fdfbd 2024-09-13T14:47:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:47:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:47:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:47:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:47:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:47:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:47:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:47:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:47:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:226789: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:47:25.665 INFO:tasks.workunit.client.0.smithi112.stderr:226789: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:47:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:47:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:47:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:47:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:47:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:47:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:47:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:47:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:47:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:47:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:47:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 1 2024-09-13T14:47:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 2 2024-09-13T14:47:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 3 2024-09-13T14:47:25.666 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 4 2024-09-13T14:47:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 5 2024-09-13T14:47:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 6 2024-09-13T14:47:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 7 2024-09-13T14:47:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 8 2024-09-13T14:47:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:226789: 9' 2024-09-13T14:47:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:47:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:47:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836592 2024-09-13T14:47:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836592 2024-09-13T14:47:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592' 2024-09-13T14:47:25.667 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.152 INFO:tasks.workunit.client.0.smithi112.stderr:2267792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:47:27.152 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673063 2024-09-13T14:47:27.153 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673063 2024-09-13T14:47:27.153 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673063' 2024-09-13T14:47:27.153 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.153 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:47:27.153 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509536 2024-09-13T14:47:27.153 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509536 2024-09-13T14:47:27.153 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673063 2-64424509536' 2024-09-13T14:47:27.153 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.153 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:47:27.153 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=506806140930 2024-09-13T14:47:27.153 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 506806140930 2024-09-13T14:47:27.153 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673063 2-64424509536 3-506806140930' 2024-09-13T14:47:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:47:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182482 2024-09-13T14:47:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182482 2024-09-13T14:47:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673063 2-64424509536 3-506806140930 4-107374182482' 2024-09-13T14:47:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:47:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=506806140931 2024-09-13T14:47:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 506806140931 2024-09-13T14:47:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673063 2-64424509536 3-506806140930 4-107374182482 5-506806140931' 2024-09-13T14:47:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:47:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855427 2024-09-13T14:47:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855427 2024-09-13T14:47:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673063 2-64424509536 3-506806140930 4-107374182482 5-506806140931 6-150323855427' 2024-09-13T14:47:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:47:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: 89: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:47:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673062 2024-09-13T14:47:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673062 2024-09-13T14:47:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673062' 2024-09-13T14:47:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:47:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509535 2024-09-13T14:47:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509535 2024-09-13T14:47:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509535' 2024-09-13T14:47:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:47:27.156 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=506806140931 2024-09-13T14:47:27.156 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 506806140931 2024-09-13T14:47:27.156 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509535 3-506806140931' 2024-09-13T14:47:27.156 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.156 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:47:27.156 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182483 2024-09-13T14:47:27.156 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182483 2024-09-13T14:47:27.156 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509535 3-506806140931 4-107374182483' 2024-09-13T14:47:27.156 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.156 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:47:27.156 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=506806140930 2024-09-13T14:47:27.156 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 506806140930 2024-09-13T14:47:27.156 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509535 3-506806140931 4-107374182483 5-506806140930' 2024-09-13T14:47:27.156 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.157 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:47:27.157 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855428 2024-09-13T14:47:27.160 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855428 2024-09-13T14:47:27.160 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509535 3-506806140931 4-107374182483 5-506806140930 6-150323855428' 2024-09-13T14:47:27.161 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.161 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:47:27.743 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fflush_pg_stats: seq=171798691903 2024-09-13T14:47:27.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691903 2024-09-13T14:47:27.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673063 2-64424509536 3-506806140930 4-107374182482 5-506806140931 6-150323855427 7-171798691903' 2024-09-13T14:47:27.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:47:27.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528372 2024-09-13T14:47:27.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528372 2024-09-13T14:47:27.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673063 2-64424509536 3-506806140930 4-107374182482 5-506806140931 6-150323855427 7-171798691903 8-193273528372' 2024-09-13T14:47:27.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:47:27.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533385 2024-09-13T14:47:27.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533385 2024-09-13T14:47:27.744 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673063 2-64424509536 3-506806140930 4-107374182482 5-506806140931 6-150323855427 7-171798691903 8-193273528372 9-455266533385' 2024-09-13T14:47:27.745 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:27.745 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836591 2024-09-13T14:47:27.745 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:27.745 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:47:27.745 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836591 2024-09-13T14:47:27.745 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:27.745 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836591 2024-09-13T14:47:27.745 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2024-09-13T14:47:27.745 INFO:tasks.workunit.client.0.smithi112.stderr:226792: waiting osd.0 seq 21474836591 2024-09-13T14:47:27.745 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:47:27.745 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836592 -lt 21474836591 2024-09-13T14:47:27.745 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:27.745 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:27.745 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673063 2024-09-13T14:47:27.745 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:47:27.746 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:27.746 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673063 2024-09-13T14:47:27.746 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673063 2024-09-13T14:47:27.746 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2024-09-13T14:47:27.746 INFO:tasks.workunit.client.0.smithi112.stderr:226792: waiting osd.1 seq 42949673063 2024-09-13T14:47:27.758 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: fluslush_pg_stats: seq=171798691904 2024-09-13T14:47:27.758 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691904 2024-09-13T14:47:27.758 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509535 3-506806140931 4-107374182483 5-506806140930 6-150323855428 7-171798691904' 2024-09-13T14:47:27.758 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.758 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:47:27.758 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528371 2024-09-13T14:47:27.759 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528371 2024-09-13T14:47:27.759 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509535 3-506806140931 4-107374182483 5-506806140930 6-150323855428 7-171798691904 8-193273528371' 2024-09-13T14:47:27.759 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:27.759 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:47:27.759 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533384 2024-09-13T14:47:27.759 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533384 2024-09-13T14:47:27.759 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673062 2-64424509535 3-506806140931 4-107374182483 5-506806140930 6-150323855428 7-171798691904 8-193273528371 9-455266533384' 2024-09-13T14:47:27.759 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:27.759 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836592 2024-09-13T14:47:27.759 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:27.759 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:47:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836592 2024-09-13T14:47:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836592 2024-09-13T14:47:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2024-09-13T14:47:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:226789: waiting osd.0 seq 21474836592 2024-09-13T14:47:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:47:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836592 -lt 21474836592 2024-09-13T14:47:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673062 2024-09-13T14:47:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:47:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673062 2024-09-13T14:47:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673062 2024-09-13T14:47:27.760 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-09-13T14:47:27.761 INFO:tasks.workunit.client.0.smithi112.stderr:226789: waiting osd.1 seq 42949673062 2024-09-13T14:47:29.058 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flushh_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:47:29.058 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673063 -lt 42949673063 2024-09-13T14:47:29.058 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:29.058 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509536 2024-09-13T14:47:29.058 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:29.058 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:47:29.058 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509536 2024-09-13T14:47:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509536 2024-09-13T14:47:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-09-13T14:47:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:226792: waiting osd.2 seq 64424509536 2024-09-13T14:47:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:47:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509537 -lt 64424509536 2024-09-13T14:47:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-506806140930 2024-09-13T14:47:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:47:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-506806140930 2024-09-13T14:47:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=506806140930 2024-09-13T14:47:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 506806140930' 2024-09-13T14:47:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:226792: waiting osd.3 seq 506806140930 2024-09-13T14:47:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:47:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 506806140931 -lt 506806140930 2024-09-13T14:47:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182482 2024-09-13T14:47:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:47:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182482 2024-09-13T14:47:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182482 2024-09-13T14:47:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182482' 2024-09-13T14:47:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:226792: waiting osd.4 seq 107374182482 2024-09-13T14:47:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:47:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182484 -lt 107374182482 2024-09-13T14:47:29.061 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:29.061 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-506806140931 2024-09-13T14:47:29.136 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubu_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:47:29.137 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673063 -lt 42949673062 2024-09-13T14:47:29.137 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:29.137 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509535 2024-09-13T14:47:29.137 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:29.137 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:47:29.137 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509535 2024-09-13T14:47:29.137 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:29.137 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509535 2024-09-13T14:47:29.137 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509535' 2024-09-13T14:47:29.137 INFO:tasks.workunit.client.0.smithi112.stderr:226789: waiting osd.2 seq 64424509535 2024-09-13T14:47:29.137 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:47:29.137 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509537 -lt 64424509535 2024-09-13T14:47:29.138 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:29.138 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-506806140931 2024-09-13T14:47:29.138 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:29.138 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:47:29.138 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-506806140931 2024-09-13T14:47:29.138 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:29.138 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=506806140931 2024-09-13T14:47:29.138 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 506806140931' 2024-09-13T14:47:29.138 INFO:tasks.workunit.client.0.smithi112.stderr:226789: waiting osd.3 seq 506806140931 2024-09-13T14:47:29.138 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:47:29.138 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 506806140931 -lt 506806140931 2024-09-13T14:47:29.138 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:29.138 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182483 2024-09-13T14:47:29.138 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:29.139 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:47:29.139 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182483 2024-09-13T14:47:29.139 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:29.139 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182483 2024-09-13T14:47:29.139 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182483' 2024-09-13T14:47:29.139 INFO:tasks.workunit.client.0.smithi112.stderr:226789: waiting osd.4 seq 107374182483 2024-09-13T14:47:29.139 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:47:29.139 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182484 -lt 107374182483 2024-09-13T14:47:29.139 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:29.139 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-506806140930 2024-09-13T14:47:30.016 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubunntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:30.016 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:47:30.016 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-506806140931 2024-09-13T14:47:30.016 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:30.016 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=506806140931 2024-09-13T14:47:30.016 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 506806140931' 2024-09-13T14:47:30.017 INFO:tasks.workunit.client.0.smithi112.stderr:226792: waiting osd.5 seq 506806140931 2024-09-13T14:47:30.017 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:47:30.017 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 506806140931 -lt 506806140931 2024-09-13T14:47:30.017 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:30.017 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855427 2024-09-13T14:47:30.017 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:30.017 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:47:30.017 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855427 2024-09-13T14:47:30.017 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:30.017 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855427 2024-09-13T14:47:30.017 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855427' 2024-09-13T14:47:30.017 INFO:tasks.workunit.client.0.smithi112.stderr:226792: waiting osd.6 seq 150323855427 2024-09-13T14:47:30.017 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:47:30.018 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855428 -lt 150323855427 2024-09-13T14:47:30.018 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:30.018 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:30.018 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691903 2024-09-13T14:47:30.018 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:47:30.018 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691903 2024-09-13T14:47:30.018 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:30.018 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691903 2024-09-13T14:47:30.018 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691903' 2024-09-13T14:47:30.018 INFO:tasks.workunit.client.0.smithi112.stderr:226792: waiting osd.7 seq 171798691903 2024-09-13T14:47:30.018 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:47:30.018 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691904 -lt 171798691903 2024-09-13T14:47:30.018 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:30.019 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528372 2024-09-13T14:47:30.019 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:30.019 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:47:30.019 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528372 2024-09-13T14:47:30.164 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_statu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:30.164 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:47:30.164 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-506806140930 2024-09-13T14:47:30.164 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:30.164 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=506806140930 2024-09-13T14:47:30.164 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 506806140930' 2024-09-13T14:47:30.164 INFO:tasks.workunit.client.0.smithi112.stderr:226789: waiting osd.5 seq 506806140930 2024-09-13T14:47:30.164 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:47:30.164 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 506806140931 -lt 506806140930 2024-09-13T14:47:30.164 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:30.164 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855428 2024-09-13T14:47:30.165 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:30.165 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:47:30.165 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855428 2024-09-13T14:47:30.165 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:30.165 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855428 2024-09-13T14:47:30.165 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855428' 2024-09-13T14:47:30.165 INFO:tasks.workunit.client.0.smithi112.stderr:226789: waiting osd.6 seq 150323855428 2024-09-13T14:47:30.165 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:47:30.165 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855428 -lt 150323855428 2024-09-13T14:47:30.165 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:30.165 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691904 2024-09-13T14:47:30.165 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:30.165 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:47:30.166 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691904 2024-09-13T14:47:30.166 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:30.166 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691904 2024-09-13T14:47:30.166 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691904' 2024-09-13T14:47:30.166 INFO:tasks.workunit.client.0.smithi112.stderr:226789: waiting osd.7 seq 171798691904 2024-09-13T14:47:30.166 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:47:30.166 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691904 -lt 171798691904 2024-09-13T14:47:30.166 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:30.166 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528371 2024-09-13T14:47:30.166 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:30.166 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:47:30.166 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528371 2024-09-13T14:47:31.798 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_statts: cut -d - -f 2 2024-09-13T14:47:31.798 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528372 2024-09-13T14:47:31.798 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528372' 2024-09-13T14:47:31.798 INFO:tasks.workunit.client.0.smithi112.stderr:226792: waiting osd.8 seq 193273528372 2024-09-13T14:47:31.798 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:47:31.798 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528372 -lt 193273528372 2024-09-13T14:47:31.799 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:31.799 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-455266533385 2024-09-13T14:47:31.799 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:31.799 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:47:31.799 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-455266533385 2024-09-13T14:47:31.799 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:31.799 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533385 2024-09-13T14:47:31.799 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 455266533385' 2024-09-13T14:47:31.799 INFO:tasks.workunit.client.0.smithi112.stderr:226792: waiting osd.9 seq 455266533385 2024-09-13T14:47:31.799 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:47:31.799 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533386 -lt 455266533385 2024-09-13T14:47:31.799 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:47:31.800 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:47:31.800 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:47:31.800 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:47:31.800 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:47:31.800 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:47:31.800 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:47:31.800 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:47:31.800 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:47:31.800 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:47:31.800 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:47:31.800 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:47:31.800 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:47:31.801 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:47:31.801 INFO:tasks.workunit.client.0.smithi112.stderr:226792: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:47:31.801 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:47:31.801 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:47:31.801 INFO:tasks.workunit.client.0.smithi112.stderr:226792: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:47:31.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:47:31.954 INFO:tasks.workunit.client.0.smithi112.stderr:s: cut -d - -f 2 2024-09-13T14:47:31.954 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528371 2024-09-13T14:47:31.954 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528371' 2024-09-13T14:47:31.954 INFO:tasks.workunit.client.0.smithi112.stderr:226789: waiting osd.8 seq 193273528371 2024-09-13T14:47:31.955 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:47:31.955 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528372 -lt 193273528371 2024-09-13T14:47:31.955 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:31.955 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-455266533384 2024-09-13T14:47:31.955 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:31.955 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:47:31.955 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-455266533384 2024-09-13T14:47:31.955 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:31.955 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533384 2024-09-13T14:47:31.955 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 455266533384' 2024-09-13T14:47:31.955 INFO:tasks.workunit.client.0.smithi112.stderr:226789: waiting osd.9 seq 455266533384 2024-09-13T14:47:31.955 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:47:31.955 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533386 -lt 455266533384 2024-09-13T14:47:31.956 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:47:31.956 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:47:31.956 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:47:31.956 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:47:31.956 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:47:31.956 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:47:31.956 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:47:31.956 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:47:31.956 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:47:31.956 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:47:31.956 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:47:31.956 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:47:31.956 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:47:31.957 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:47:31.957 INFO:tasks.workunit.client.0.smithi112.stderr:226789: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:47:31.957 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:47:31.957 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:47:31.957 INFO:tasks.workunit.client.0.smithi112.stderr:226789: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:47:31.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:47:31.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:47:31.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 226788 2024-09-13T14:47:31.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:47:31.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:47:31.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:47:31.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-13T14:47:31.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:47:31.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-13T14:47:31.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:47:31.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:47:31.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:47:31.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:47:32.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-13T14:47:32.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-13T14:47:32.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:47:32.265 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:47:32.265 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:47:32.265 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:47:32.265 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:47:32.265 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:47:32.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:46:52.809632+0000 2024-09-13T14:47:32.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:47:32.897 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:47:32.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:46:52.809632+0000 2024-09-13T14:47:32.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:47:32.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:46:52.809632+0000 2024-09-13T14:47:32.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:47:32.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:47:32.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:47:32.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:47:32.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:47:32.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:47:32.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:47:32.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:47:33.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:46:52.809632+0000 '>' 2024-09-13T14:46:52.809632+0000 2024-09-13T14:47:33.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:47:34.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:47:34.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:47:34.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:47:34.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:47:34.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:47:34.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:47:34.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:47:34.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:46:52.809632+0000 '>' 2024-09-13T14:46:52.809632+0000 2024-09-13T14:47:34.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:47:35.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:47:35.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:47:35.552 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:47:35.552 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:47:35.552 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:47:35.553 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:47:35.553 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:47:35.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:46:52.809632+0000 '>' 2024-09-13T14:46:52.809632+0000 2024-09-13T14:47:35.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:47:36.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:47:36.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:47:36.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:47:36.878 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:47:36.878 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:47:36.878 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:47:36.878 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:47:37.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:47:33.644764+0000 '>' 2024-09-13T14:46:52.809632+0000 2024-09-13T14:47:37.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:47:37.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:47:39.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-13T14:47:39.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:47:39.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:47:39.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:47:39.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 230236"' 2024-09-13T14:47:39.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 230236' 2024-09-13T14:47:39.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:47:39.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:47:39.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:47:39.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:47:39.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 230238"' 2024-09-13T14:47:39.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 230238' 2024-09-13T14:47:39.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-13T14:47:39.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/230239: /' 2024-09-13T14:47:39.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 230236 230238' 2024-09-13T14:47:39.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:47:39.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:47:39.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 230236 2024-09-13T14:47:39.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:47:39.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/230242: /' 2024-09-13T14:47:40.453 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:47:40.453 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:47:40.453 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:47:40.453 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:47:40.453 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:47:40.453 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:47:40.453 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:47:40.453 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:47:40.454 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:47:40.454 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:47:40.454 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:47:40.454 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:47:40.454 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:47:40.454 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:47:40.454 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:47:40.454 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:47:40.454 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:47:40.454 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:47:40.454 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:47:40.454 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:47:40.454 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:47:40.455 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:47:40.455 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-13T14:47:40.455 INFO:tasks.workunit.client.0.smithi112.stderr:230239: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-13T14:47:40.455 INFO:tasks.workunit.client.0.smithi112.stderr:230239: _ 2024-09-13T14:47:40.455 INFO:tasks.workunit.client.0.smithi112.stderr:230239: hinfo_key 2024-09-13T14:47:40.455 INFO:tasks.workunit.client.0.smithi112.stderr:230239: snapset 2024-09-13T14:47:40.455 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:47:40.455 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:47:40.455 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:47:40.455 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:47:40.455 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:47:40.455 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:47:40.455 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:47:40.458 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:47:40.459 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:47:40.459 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:47:40.459 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:47:40.459 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:47:40.459 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:47:40.459 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:47:40.459 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:47:40.459 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:47:40.459 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:47:40.459 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:47:40.459 INFO:tasks.workunit.client.0.smithi112.stderr:230239: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:47:40.459 INFO:tasks.workunit.client.0.smithi112.stderr:230239: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:47:40.459 INFO:tasks.workunit.client.0.smithi112.stderr:230239: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:47:40.459 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:47:40.460 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:47:40.460 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:47:40.460 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:47:40.460 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:47:40.460 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:47:40.460 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:47:40.460 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:47:40.460 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:47:40.460 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:47:40.460 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:47:40.460 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:47:40.460 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:47:40.460 INFO:tasks.workunit.client.0.smithi112.stderr:230239: start osd.3 2024-09-13T14:47:40.530 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-l230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:47:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:47:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-13T14:47:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:47:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:47:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:47:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:47:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-13T14:47:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:47:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-13T14:47:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:47:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:47:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:47:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:47:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:47:40.531 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:47:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:47:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:47:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:47:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-13T14:47:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:47:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:47:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-13T14:47:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:230242: _ 2024-09-13T14:47:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:230242: hinfo_key 2024-09-13T14:47:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:230242: snapset 2024-09-13T14:47:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-13T14:47:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:47:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:47:40.532 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-13T14:47:40.533 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:47:40.533 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:47:40.533 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:47:40.533 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:47:40.533 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:47:40.533 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:47:40.533 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-13T14:47:40.533 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-13T14:47:40.533 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:47:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:47:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:47:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:47:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:47:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:47:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:230242: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:47:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:230242: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:47:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:230242: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:47:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:47:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:47:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:47:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:47:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:47:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:47:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:47:40.535 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:47:40.535 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:47:40.535 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:47:40.535 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:47:40.535 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-13T14:47:40.535 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-13T14:47:40.535 INFO:tasks.workunit.client.0.smithi112.stderr:230242: start osd.5 2024-09-13T14:47:45.204 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recoen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:47:45.204 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:47:45.204 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:47:45.205 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:47:45.205 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:47:45.205 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:47:45.205 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 2024-09-13T14:47:40.499+0000 7f43c6afa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:45.205 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 2024-09-13T14:47:40.539+0000 7f43c6afa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:45.205 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 2024-09-13T14:47:40.539+0000 7f43c6afa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:45.205 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:47:45.205 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:47:45.205 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:47:45.205 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:47:45.205 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:47:45.205 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:45.205 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:47:45.206 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 0 2024-09-13T14:47:45.206 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:45.206 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:47:45.206 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:45.206 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 2024-09-13T14:47:41.627+0000 7f43c6afa7c0 -1 Falling back to public interface 2024-09-13T14:47:45.206 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:45.206 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:45.206 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:47:45.206 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 1 2024-09-13T14:47:45.206 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:45.206 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:47:45.206 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:45.206 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:45.206 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:45.206 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:47:45.207 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 2 2024-09-13T14:47:45.207 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:45.207 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:47:45.207 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:45.207 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:45.207 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:45.207 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:47:45.207 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 3 2024-09-13T14:47:45.207 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:45.207 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:47:46.043 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.very_ops 2024-09-13T14:47:46.043 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-13T14:47:46.043 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-13T14:47:46.043 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:47:46.043 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:47:46.043 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:47:46.043 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 2024-09-13T14:47:40.563+0000 7fa5ecb1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:46.043 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 2024-09-13T14:47:40.563+0000 7fa5ecb1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:46.043 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 2024-09-13T14:47:40.567+0000 7fa5ecb1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:46.043 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-13T14:47:46.044 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:47:46.044 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-13T14:47:46.044 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:47:46.044 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:47:46.044 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:46.044 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:47:46.044 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 0 2024-09-13T14:47:46.044 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:46.044 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:46.044 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:46.044 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 2024-09-13T14:47:41.927+0000 7fa5ecb1b7c0 -1 Falling back to public interface 2024-09-13T14:47:46.044 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:46.044 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:46.045 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:47:46.045 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 1 2024-09-13T14:47:46.045 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:46.045 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:46.045 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:46.045 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:46.045 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:46.045 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:47:46.045 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 2 2024-09-13T14:47:46.045 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:46.045 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:46.045 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:46.045 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:46.046 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:46.046 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:47:46.046 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 3 2024-09-13T14:47:46.046 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:46.046 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:46.046 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:47.189 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 2024-09-13T14:47:46.043+0000 7fa50/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:47.190 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 2024-09-13T14:47:45.455+0000 7f43c6afa7c0 -1 osd.3 119 log_to_monitors true 2024-09-13T14:47:47.190 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:47.190 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:47.190 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:47:47.190 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 4 2024-09-13T14:47:47.190 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:47.190 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:47:47.190 INFO:tasks.workunit.client.0.smithi112.stderr:230239: osd.3 up in weight 1 up_from 123 up_thru 118 down_at 120 last_clean_interval [118,119) [v2:127.0.0.1:6826/4213980024,v1:127.0.0.1:6827/4213980024] [v2:127.0.0.1:6828/4213980024,v1:127.0.0.1:6829/4213980024] exists,up 240beeea-9a32-48d8-9b95-819a8b7fdfbd 2024-09-13T14:47:47.190 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:47:47.190 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:47:47.190 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:47:47.190 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:47:47.191 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:47:47.191 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:47:47.191 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:47:47.191 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:47:47.191 INFO:tasks.workunit.client.0.smithi112.stderr:230239: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:47:47.191 INFO:tasks.workunit.client.0.smithi112.stderr:230239: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:47:47.191 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:47:47.191 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:47:47.191 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:47:47.191 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:47:47.191 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:47:47.191 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:47:47.191 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:47:47.191 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:47:47.192 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:47:47.192 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:47:47.192 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 1 2024-09-13T14:47:47.192 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 2 2024-09-13T14:47:47.192 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 3 2024-09-13T14:47:47.192 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 4 2024-09-13T14:47:47.192 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 5 2024-09-13T14:47:47.192 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 6 2024-09-13T14:47:47.192 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 7 2024-09-13T14:47:47.192 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 8 2024-09-13T14:47:47.192 INFO:tasks.workunit.client.0.smithi112.stderr:230239: 9' 2024-09-13T14:47:47.192 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:47:47.192 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:47.192 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:47:47.193 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836597 2024-09-13T14:47:47.193 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836597 2024-09-13T14:47:47.193 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597' 2024-09-13T14:47:49.076 INFO:tasks.workunit.client.0.smithi112.stderr:230239ecb1b7c0 -1 osd.5 119 log_to_monitors true 2024-09-13T14:47:49.076 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:49.076 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:49.076 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:47:49.076 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 4 2024-09-13T14:47:49.077 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:49.077 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:49.077 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:47:49.077 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:47:49.077 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:47:49.077 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:47:49.077 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 5 2024-09-13T14:47:49.077 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:47:49.077 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:47:49.077 INFO:tasks.workunit.client.0.smithi112.stderr:230242: osd.5 up in weight 1 up_from 124 up_thru 124 down_at 120 last_clean_interval [118,119) [v2:127.0.0.1:6874/2206351717,v1:127.0.0.1:6875/2206351717] [v2:127.0.0.1:6876/2206351717,v1:127.0.0.1:6877/2206351717] exists,up ac5b44cf-f8cb-4d05-8d84-3858f28ea531 2024-09-13T14:47:49.077 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:47:49.077 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:47:49.077 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:47:49.078 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:47:49.078 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:47:49.078 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:47:49.078 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:47:49.078 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:47:49.078 INFO:tasks.workunit.client.0.smithi112.stderr:230242: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:47:49.078 INFO:tasks.workunit.client.0.smithi112.stderr:230242: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:47:49.078 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:47:49.078 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:47:49.078 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:47:49.078 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:47:49.078 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:47:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:47:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:47:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:47:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:47:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:47:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 1 2024-09-13T14:47:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 2 2024-09-13T14:47:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 3 2024-09-13T14:47:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 4 2024-09-13T14:47:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 5 2024-09-13T14:47:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 6 2024-09-13T14:47:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 7 2024-09-13T14:47:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 8 2024-09-13T14:47:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:230242: 9' 2024-09-13T14:47:49.080 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:47:49.085 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standa: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:49.086 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:47:49.086 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673068 2024-09-13T14:47:49.086 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673068 2024-09-13T14:47:49.086 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673068' 2024-09-13T14:47:49.086 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:49.086 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:47:49.086 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509542 2024-09-13T14:47:49.086 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509542 2024-09-13T14:47:49.086 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673068 2-64424509542' 2024-09-13T14:47:49.086 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:49.086 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:47:49.086 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=528280977410 2024-09-13T14:47:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 528280977410 2024-09-13T14:47:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673068 2-64424509542 3-528280977410' 2024-09-13T14:47:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:47:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182489 2024-09-13T14:47:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182489 2024-09-13T14:47:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673068 2-64424509542 3-528280977410 4-107374182489' 2024-09-13T14:47:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:47:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=532575944706 2024-09-13T14:47:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 532575944706 2024-09-13T14:47:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673068 2-64424509542 3-528280977410 4-107374182489 5-532575944706' 2024-09-13T14:47:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:49.088 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:47:49.088 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855434 2024-09-13T14:47:49.088 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855434 2024-09-13T14:47:49.088 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673068 2-64424509542 3-528280977410 4-107374182489 5-532575944706 6-150323855434' 2024-09-13T14:47:49.088 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:49.910 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:47:49.910 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691909 2024-09-13T14:47:49.911 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691909 2024-09-13T14:47:49.911 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673068 2-64424509542 3-528280977410 4-107374182489 5-532575944706 6-150323855434 7-171798691909' 2024-09-13T14:47:49.911 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:49.911 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:47:49.911 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528377 2024-09-13T14:47:49.911 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528377 2024-09-13T14:47:49.911 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673068 2-64424509542 3-528280977410 4-107374182489 5-532575944706 6-150323855434 7-171798691909 8-193273528377' 2024-09-13T14:47:49.911 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:49.911 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:47:49.911 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533391 2024-09-13T14:47:49.912 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533391 2024-09-13T14:47:49.912 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673068 2-64424509542 3-528280977410 4-107374182489 5-532575944706 6-150323855434 7-171798691909 8-193273528377 9-455266533391' 2024-09-13T14:47:49.912 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:49.912 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836597 2024-09-13T14:47:49.912 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:49.912 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:47:49.912 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836597 2024-09-13T14:47:49.912 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:49.912 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836597 2024-09-13T14:47:49.912 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836597' 2024-09-13T14:47:49.912 INFO:tasks.workunit.client.0.smithi112.stderr:230239: waiting osd.0 seq 21474836597 2024-09-13T14:47:49.912 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:47:49.912 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836598 -lt 21474836597 2024-09-13T14:47:49.912 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:49.913 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673068 2024-09-13T14:47:49.913 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:49.913 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:47:49.913 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673068 2024-09-13T14:47:49.913 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:49.913 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673068 2024-09-13T14:47:49.913 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673068' 2024-09-13T14:47:50.161 INFO:tasks.workunit.client.0.smithi112.stderr:230239: waitilone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:50.161 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:47:50.161 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836599 2024-09-13T14:47:50.161 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836599 2024-09-13T14:47:50.161 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599' 2024-09-13T14:47:50.161 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:50.161 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:47:50.161 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673070 2024-09-13T14:47:50.162 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673070 2024-09-13T14:47:50.162 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673070' 2024-09-13T14:47:50.162 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:50.162 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:47:50.162 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509543 2024-09-13T14:47:50.162 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509543 2024-09-13T14:47:50.162 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673070 2-64424509543' 2024-09-13T14:47:50.162 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:50.162 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:47:50.162 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=528280977411 2024-09-13T14:47:50.162 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 528280977411 2024-09-13T14:47:50.162 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673070 2-64424509543 3-528280977411' 2024-09-13T14:47:50.163 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:50.163 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:47:50.163 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182490 2024-09-13T14:47:50.163 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182490 2024-09-13T14:47:50.163 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673070 2-64424509543 3-528280977411 4-107374182490' 2024-09-13T14:47:50.163 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:50.163 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:47:50.163 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=532575944707 2024-09-13T14:47:50.163 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 532575944707 2024-09-13T14:47:50.163 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673070 2-64424509543 3-528280977411 4-107374182490 5-532575944707' 2024-09-13T14:47:50.163 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:50.163 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:47:50.939 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855435 2024-09-13T14:47:50.939 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855435 2024-09-13T14:47:50.939 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673070 2-64424509543 3-528280977411 4-107374182490 5-532575944707 6-150323855435' 2024-09-13T14:47:50.940 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:50.940 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:47:50.940 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691910 2024-09-13T14:47:50.940 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691910 2024-09-13T14:47:50.940 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673070 2-64424509543 3-528280977411 4-107374182490 5-532575944707 6-150323855435 7-171798691910' 2024-09-13T14:47:50.940 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:50.940 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:47:50.940 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528378 2024-09-13T14:47:50.940 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528378 2024-09-13T14:47:50.940 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673070 2-64424509543 3-528280977411 4-107374182490 5-532575944707 6-150323855435 7-171798691910 8-193273528378' 2024-09-13T14:47:50.940 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:47:50.940 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:47:50.941 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533392 2024-09-13T14:47:50.941 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533392 2024-09-13T14:47:50.941 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673070 2-64424509543 3-528280977411 4-107374182490 5-532575944707 6-150323855435 7-171798691910 8-193273528378 9-455266533392' 2024-09-13T14:47:50.941 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:50.941 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836599 2024-09-13T14:47:50.941 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:50.941 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:47:50.941 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836599 2024-09-13T14:47:50.941 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:50.941 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836599 2024-09-13T14:47:50.941 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836599' 2024-09-13T14:47:50.941 INFO:tasks.workunit.client.0.smithi112.stderr:230242: waiting osd.0 seq 21474836599 2024-09-13T14:47:50.941 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:47:50.941 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836599 -lt 21474836599 2024-09-13T14:47:50.942 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:50.942 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:50.942 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673070 2024-09-13T14:47:51.303 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/sng osd.1 seq 42949673068 2024-09-13T14:47:51.304 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:47:51.304 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673068 -lt 42949673068 2024-09-13T14:47:51.304 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:51.304 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509542 2024-09-13T14:47:51.304 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:51.304 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:47:51.304 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509542 2024-09-13T14:47:51.304 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:51.304 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509542 2024-09-13T14:47:51.304 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509542' 2024-09-13T14:47:51.304 INFO:tasks.workunit.client.0.smithi112.stderr:230239: waiting osd.2 seq 64424509542 2024-09-13T14:47:51.305 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:47:51.305 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509543 -lt 64424509542 2024-09-13T14:47:51.305 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:51.305 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-528280977410 2024-09-13T14:47:51.305 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:51.305 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:47:51.305 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-528280977410 2024-09-13T14:47:51.305 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:51.305 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=528280977410 2024-09-13T14:47:51.305 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 528280977410' 2024-09-13T14:47:51.305 INFO:tasks.workunit.client.0.smithi112.stderr:230239: waiting osd.3 seq 528280977410 2024-09-13T14:47:51.305 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:47:51.305 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 528280977411 -lt 528280977410 2024-09-13T14:47:51.305 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:51.306 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182489 2024-09-13T14:47:51.306 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:51.306 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:47:51.306 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182489 2024-09-13T14:47:51.306 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:51.306 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182489 2024-09-13T14:47:51.306 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182489' 2024-09-13T14:47:51.306 INFO:tasks.workunit.client.0.smithi112.stderr:230239: waiting osd.4 seq 107374182489 2024-09-13T14:47:51.306 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:47:51.306 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182490 -lt 107374182489 2024-09-13T14:47:51.307 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:51.931 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtandalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:47:51.931 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673070 2024-09-13T14:47:51.931 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:51.931 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673070 2024-09-13T14:47:51.931 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673070' 2024-09-13T14:47:51.931 INFO:tasks.workunit.client.0.smithi112.stderr:230242: waiting osd.1 seq 42949673070 2024-09-13T14:47:51.931 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:47:51.931 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673070 -lt 42949673070 2024-09-13T14:47:51.931 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:51.932 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509543 2024-09-13T14:47:51.932 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:51.932 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:47:51.932 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509543 2024-09-13T14:47:51.932 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:51.932 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509543 2024-09-13T14:47:51.932 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509543' 2024-09-13T14:47:51.932 INFO:tasks.workunit.client.0.smithi112.stderr:230242: waiting osd.2 seq 64424509543 2024-09-13T14:47:51.932 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:47:51.932 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509543 -lt 64424509543 2024-09-13T14:47:51.932 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:51.932 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:51.932 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-528280977411 2024-09-13T14:47:51.932 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:47:51.933 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-528280977411 2024-09-13T14:47:51.933 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:51.933 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=528280977411 2024-09-13T14:47:51.933 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 528280977411' 2024-09-13T14:47:51.933 INFO:tasks.workunit.client.0.smithi112.stderr:230242: waiting osd.3 seq 528280977411 2024-09-13T14:47:51.933 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:47:51.933 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 528280977411 -lt 528280977411 2024-09-13T14:47:51.933 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:51.933 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182490 2024-09-13T14:47:51.933 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:51.933 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:47:51.933 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182490 2024-09-13T14:47:51.933 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:51.933 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182490 2024-09-13T14:47:52.272 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cetest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-532575944706 2024-09-13T14:47:52.272 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:52.272 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:47:52.272 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-532575944706 2024-09-13T14:47:52.272 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:52.272 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=532575944706 2024-09-13T14:47:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 532575944706' 2024-09-13T14:47:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:230239: waiting osd.5 seq 532575944706 2024-09-13T14:47:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:47:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 532575944707 -lt 532575944706 2024-09-13T14:47:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855434 2024-09-13T14:47:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:47:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855434 2024-09-13T14:47:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855434 2024-09-13T14:47:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855434' 2024-09-13T14:47:52.273 INFO:tasks.workunit.client.0.smithi112.stderr:230239: waiting osd.6 seq 150323855434 2024-09-13T14:47:52.274 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:47:52.274 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855435 -lt 150323855434 2024-09-13T14:47:52.274 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:52.274 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691909 2024-09-13T14:47:52.274 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:52.274 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:47:52.274 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691909 2024-09-13T14:47:52.274 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:52.274 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691909 2024-09-13T14:47:52.274 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691909' 2024-09-13T14:47:52.274 INFO:tasks.workunit.client.0.smithi112.stderr:230239: waiting osd.7 seq 171798691909 2024-09-13T14:47:52.274 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:47:52.274 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691909 -lt 171798691909 2024-09-13T14:47:52.274 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:52.274 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528377 2024-09-13T14:47:52.275 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:52.275 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:47:53.277 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182490' 2024-09-13T14:47:53.277 INFO:tasks.workunit.client.0.smithi112.stderr:230242: waiting osd.4 seq 107374182490 2024-09-13T14:47:53.277 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:47:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182490 -lt 107374182490 2024-09-13T14:47:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-532575944707 2024-09-13T14:47:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:47:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-532575944707 2024-09-13T14:47:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=532575944707 2024-09-13T14:47:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 532575944707' 2024-09-13T14:47:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:230242: waiting osd.5 seq 532575944707 2024-09-13T14:47:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:47:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 532575944708 -lt 532575944707 2024-09-13T14:47:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855435 2024-09-13T14:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855435 2024-09-13T14:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855435 2024-09-13T14:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855435' 2024-09-13T14:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:230242: waiting osd.6 seq 150323855435 2024-09-13T14:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855435 -lt 150323855435 2024-09-13T14:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691910 2024-09-13T14:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691910 2024-09-13T14:47:53.280 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:53.280 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691910 2024-09-13T14:47:53.280 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691910' 2024-09-13T14:47:53.280 INFO:tasks.workunit.client.0.smithi112.stderr:230242: waiting osd.7 seq 171798691910 2024-09-13T14:47:53.280 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:47:53.280 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691911 -lt 171798691910 2024-09-13T14:47:54.145 INFO:tasks.workunit.client.0.smithi112.stderr:230o 8-193273528377 2024-09-13T14:47:54.145 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:54.145 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528377 2024-09-13T14:47:54.145 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528377' 2024-09-13T14:47:54.145 INFO:tasks.workunit.client.0.smithi112.stderr:230239: waiting osd.8 seq 193273528377 2024-09-13T14:47:54.145 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:47:54.145 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528379 -lt 193273528377 2024-09-13T14:47:54.145 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:54.145 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-455266533391 2024-09-13T14:47:54.145 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:54.145 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:47:54.145 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-455266533391 2024-09-13T14:47:54.146 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:54.146 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533391 2024-09-13T14:47:54.146 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 455266533391' 2024-09-13T14:47:54.146 INFO:tasks.workunit.client.0.smithi112.stderr:230239: waiting osd.9 seq 455266533391 2024-09-13T14:47:54.146 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:47:54.146 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533392 -lt 455266533391 2024-09-13T14:47:54.146 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:47:54.146 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:47:54.146 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:47:54.146 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:47:54.146 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:47:54.146 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:47:54.146 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:47:54.146 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:47:54.147 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:47:54.147 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:47:54.147 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:47:54.147 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:47:54.147 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:47:54.147 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:47:54.147 INFO:tasks.workunit.client.0.smithi112.stderr:230239: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:47:54.147 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:47:54.147 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:47:54.147 INFO:tasks.workunit.client.0.smithi112.stderr:230239: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:47:54.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:47:54.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:47:54.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 230238 2024-09-13T14:47:54.794 INFO:tasks.workunit.client.0.smithi112.stderr:242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:54.794 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:54.794 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528378 2024-09-13T14:47:54.794 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:47:54.794 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528378 2024-09-13T14:47:54.794 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:54.794 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528378 2024-09-13T14:47:54.794 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528378' 2024-09-13T14:47:54.795 INFO:tasks.workunit.client.0.smithi112.stderr:230242: waiting osd.8 seq 193273528378 2024-09-13T14:47:54.795 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:47:54.795 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528379 -lt 193273528378 2024-09-13T14:47:54.795 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:47:54.795 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-455266533392 2024-09-13T14:47:54.795 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:47:54.795 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:47:54.795 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-455266533392 2024-09-13T14:47:54.795 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:47:54.795 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533392 2024-09-13T14:47:54.795 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 455266533392' 2024-09-13T14:47:54.795 INFO:tasks.workunit.client.0.smithi112.stderr:230242: waiting osd.9 seq 455266533392 2024-09-13T14:47:54.795 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:47:54.795 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533392 -lt 455266533392 2024-09-13T14:47:54.796 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:47:54.796 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:47:54.796 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:47:54.796 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:47:54.796 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:47:54.796 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:47:54.796 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:47:54.796 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:47:54.796 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:47:54.796 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:47:54.796 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:47:54.796 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:47:55.169 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:47:55.170 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:47:55.170 INFO:tasks.workunit.client.0.smithi112.stderr:230242: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:47:55.170 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:47:55.170 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:47:55.170 INFO:tasks.workunit.client.0.smithi112.stderr:230242: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:47:55.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:47:55.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:47:55.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:47:55.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:47:55.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-13T14:47:55.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:47:55.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:47:55.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:47:55.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T14:47:55.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:47:55.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:47:55.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:47:55.249 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:47:55.249 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:47:55.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:47:55.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:47:55.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:47:55.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:47:55.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:47:55.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:47:55.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:47:55.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:47:55.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:47:55.570 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:47:55.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:47:55.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:47:55.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:47:55.572 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:47:55.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:47:55.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:47:55.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:47:55.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:47:55.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:47:55.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:47:55.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:47:55.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:47:55.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:47:55.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T14:47:55.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T14:47:55.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T14:47:55.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T14:47:55.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:47:55.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:47:55.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:47:55.649 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:47:55.649 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:47:55.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:47:55.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:47:55.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:47:55.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:47:55.652 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:47:55.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:47:55.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:47:55.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:47:55.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:47:55.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:47:55.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:47:55.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:47:55.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:47:55.656 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:47:55.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:47:55.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:47:55.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:47:55.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:47:55.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:47:55.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:47:55.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:47:55.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:47:55.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:47:55.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T14:47:55.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:47:55.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:47:55.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:47:55.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T14:47:55.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:47:55.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T14:47:55.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:47:55.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T14:47:55.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-09-13T14:47:55.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-09-13T14:47:55.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-09-13T14:47:55.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-09-13T14:47:55.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-09-13T14:47:55.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-09-13T14:47:55.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-09-13T14:47:55.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T14:47:55.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T14:47:55.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T14:47:55.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T14:47:55.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T14:47:55.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T14:47:56.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T14:47:56.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:47:56.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:47:56.117 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:47:56.117 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:47:56.117 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:47:56.117 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:47:56.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T14:47:56.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T14:47:56.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T14:47:56.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:47:56.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:47:56.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T14:47:56.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T14:47:56.330 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:47:56.330 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:47:56.330 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:47:56.331 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:47:56.331 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:47:56.331 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:47:56.331 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:47:56.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:47:56.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T14:47:56.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T14:47:56.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:47:56.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:47:56.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T14:47:56.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T14:47:56.451 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:47:56.451 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:47:56.451 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:47:56.451 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:47:56.451 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:47:56.451 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:47:56.452 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:47:56.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:47:56.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T14:47:56.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-09-13T14:47:56.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T14:47:56.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T14:47:56.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T14:47:56.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T14:47:56.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T14:47:56.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T14:47:57.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T14:47:57.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:47:57.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:47:57.393 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:47:57.393 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:47:57.393 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:47:57.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:47:57.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:47:57.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:47:57.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-09-13T14:47:57.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:47:57.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-09-13T14:47:57.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:47:57.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:47:57.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T14:47:57.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:47:57.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:47:57.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:47:57.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:47:57.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:47:57.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:47:57.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:47:57.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:47:57.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:47:57.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:47:57.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:47:57.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:47:57.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:47:57.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:47:57.436 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:47:57.436 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:47:57.436 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:47:57.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:47:57.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:47:57.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:47:57.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:47:57.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:47:57.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:47:57.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:47:57.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:47:57.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:47:57.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:47:57.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:47:57.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:47:57.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:47:57.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:47:57.438 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:47:57.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7ff1a9e1-5d32-4329-89aa-e288a761292f 2024-09-13T14:47:57.438 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 7ff1a9e1-5d32-4329-89aa-e288a761292f 2024-09-13T14:47:57.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7ff1a9e1-5d32-4329-89aa-e288a761292f' 2024-09-13T14:47:57.439 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:47:57.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCdUORmcf4rGxAApdekVQuIqVhVZfSz872z5g== 2024-09-13T14:47:57.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCdUORmcf4rGxAApdekVQuIqVhVZfSz872z5g=="}' 2024-09-13T14:47:57.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7ff1a9e1-5d32-4329-89aa-e288a761292f -i td/osd-scrub-repair/0/new.json 2024-09-13T14:47:57.688 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:47:57.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T14:47:57.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdUORmcf4rGxAApdekVQuIqVhVZfSz872z5g== --osd-uuid 7ff1a9e1-5d32-4329-89aa-e288a761292f 2024-09-13T14:47:57.733 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:47:57.731+0000 7f2b86bb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:57.739 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:47:57.739+0000 7f2b86bb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:57.740 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:47:57.739+0000 7f2b86bb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:47:57.741 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:47:57.739+0000 7f2b86bb17c0 -1 bdev(0x55e514a03c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:47:57.741 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:47:57.739+0000 7f2b86bb17c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T14:48:03.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T14:48:03.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:48:03.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T14:48:03.910 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:48:03.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:48:04.327 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:48:04.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T14:48:04.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:48:04.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:48:04.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:48:04.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:48:04.362 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:48:04.358+0000 7f0888c3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:48:04.394 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:48:04.390+0000 7f0888c3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:48:04.395 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:48:04.394+0000 7f0888c3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:48:04.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T14:48:04.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:48:04.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:48:04.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:48:04.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:48:04.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:04.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:48:04.681 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:48:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:04.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:05.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:05.673 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:48:05.670+0000 7f0888c3a7c0 -1 Falling back to public interface 2024-09-13T14:48:06.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:06.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:06.024 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:48:06.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:48:06.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:06.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:06.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:07.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:07.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:07.361 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:48:07.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:48:07.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:07.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:07.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:08.615 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:48:08.614+0000 7f0888c3a7c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:48:08.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:08.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:08.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:48:08.697 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:48:08.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:08.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:09.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:10.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:10.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:10.090 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:48:10.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:48:10.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:10.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:10.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:11.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:11.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:11.426 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:48:11.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:48:11.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:11.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:11.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:12.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:12.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:12.763 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:48:12.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:48:12.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:12.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:13.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:14.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:14.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:14.098 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:48:14.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:48:14.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:14.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:14.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:15.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:15.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:15.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:48:15.572 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:48:15.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:15.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:15.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:16.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:16.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:16.910 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:48:16.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:48:16.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:16.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:17.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:18.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:18.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:18.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:48:18.241 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:48:18.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:18.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:18.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:19.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:19.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:19.573 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:48:19.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:48:19.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:19.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:20.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:21.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:21.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:21.070 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:48:21.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:48:21.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:21.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:21.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:22.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:22.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:22.474 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:48:22.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:48:22.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:22.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:23.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:24.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:24.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:24.247 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:48:24.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:48:24.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:24.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:24.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:25.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:25.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:25.678 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:48:25.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:48:25.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:25.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:26.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:27.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:27.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:27.137 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:48:27.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:48:27.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:27.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:27.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:28.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:28.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:28.469 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:48:28.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:48:28.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:28.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:28.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:29.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:29.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:29.904 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:48:29.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:48:29.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:29.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:30.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:31.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:31.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:31.247 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:48:31.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:48:31.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:31.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:48:31.579 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/591670248,v1:127.0.0.1:6803/591670248] [v2:127.0.0.1:6804/591670248,v1:127.0.0.1:6805/591670248] exists,up 7ff1a9e1-5d32-4329-89aa-e288a761292f 2024-09-13T14:48:31.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:48:31.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:48:31.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:48:31.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:48:31.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-09-13T14:48:31.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:48:31.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:48:31.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T14:48:31.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:48:31.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:48:31.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:48:31.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:48:31.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:48:31.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:48:31.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:48:31.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:48:31.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:48:31.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:48:31.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:48:31.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:48:31.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:48:31.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:48:31.584 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:48:31.584 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:48:31.584 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:48:31.584 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:48:31.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:48:31.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:48:31.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:48:31.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:48:31.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:48:31.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:48:31.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:48:31.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:48:31.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:48:31.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:48:31.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:48:31.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:48:31.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:48:31.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:48:31.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=77bdcafa-961d-40b3-b535-4143a4c7ad3b 2024-09-13T14:48:31.586 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 77bdcafa-961d-40b3-b535-4143a4c7ad3b 2024-09-13T14:48:31.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 77bdcafa-961d-40b3-b535-4143a4c7ad3b' 2024-09-13T14:48:31.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:48:31.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC/UORmLtZVJBAAh2WMSBFkX4ReaqgFmcBk5A== 2024-09-13T14:48:31.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC/UORmLtZVJBAAh2WMSBFkX4ReaqgFmcBk5A=="}' 2024-09-13T14:48:31.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 77bdcafa-961d-40b3-b535-4143a4c7ad3b -i td/osd-scrub-repair/1/new.json 2024-09-13T14:48:31.972 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:48:31.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T14:48:31.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/UORmLtZVJBAAh2WMSBFkX4ReaqgFmcBk5A== --osd-uuid 77bdcafa-961d-40b3-b535-4143a4c7ad3b 2024-09-13T14:48:32.028 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:48:32.026+0000 7fc6a83457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:48:32.049 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:48:32.046+0000 7fc6a83457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:48:32.050 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:48:32.050+0000 7fc6a83457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:48:32.051 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:48:32.050+0000 7fc6a83457c0 -1 bdev(0x556a18014800 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:48:32.051 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:48:32.050+0000 7fc6a83457c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T14:48:37.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T14:48:37.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:48:37.940 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:48:37.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T14:48:37.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:48:38.350 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:48:38.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T14:48:38.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:48:38.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:48:38.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:48:38.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:48:38.382 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:48:38.378+0000 7fd41b4277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:48:38.384 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:48:38.382+0000 7fd41b4277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:48:38.385 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:48:38.382+0000 7fd41b4277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:48:38.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T14:48:38.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:48:38.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:48:38.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:48:38.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:48:38.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:38.662 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:48:38.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:48:38.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:38.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:48:38.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:39.697 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:48:39.694+0000 7fd41b4277c0 -1 Falling back to public interface 2024-09-13T14:48:39.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:39.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:39.992 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:48:39.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:48:39.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:39.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:48:40.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:41.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:41.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:41.321 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:48:41.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:48:41.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:41.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:48:41.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:42.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:42.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:42.640 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:48:42.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:48:42.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:42.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:48:42.711 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:48:42.709+0000 7fd41b4277c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:48:43.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:44.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:44.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:44.024 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:48:44.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:48:44.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:44.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:48:44.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:45.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:45.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:45.383 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:48:45.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:48:45.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:45.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:48:45.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:46.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:46.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:46.721 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:48:46.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:48:46.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:46.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:48:47.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:48.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:48.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:48.099 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:48:48.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:48:48.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:48.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:48:48.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:49.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:49.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:49.526 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:48:49.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:48:49.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:49.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:48:49.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:50.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:50.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:50.865 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:48:50.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:48:50.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:50.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:48:51.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:52.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:52.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:52.203 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:48:52.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:48:52.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:52.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:48:52.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:53.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:53.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:53.543 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:48:53.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:48:53.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:53.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:48:53.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:54.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:54.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:54.878 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:48:54.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:48:54.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:54.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:48:55.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:56.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:56.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:56.424 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:48:56.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:48:56.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:56.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:48:57.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:58.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:58.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:58.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:48:58.074 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:48:58.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:58.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:48:58.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:48:59.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:48:59.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:48:59.509 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:48:59.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:48:59.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:48:59.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:49:00.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:01.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:01.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:01.179 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:49:01.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:49:01.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:01.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:49:01.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:02.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:02.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:02.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:49:02.940 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:49:02.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:02.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:49:03.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:04.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:04.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:04.399 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:49:04.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:49:04.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:04.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:49:05.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:06.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:06.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:06.148 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:49:06.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:49:06.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:06.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:49:06.475 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3996553280,v1:127.0.0.1:6811/3996553280] [v2:127.0.0.1:6812/3996553280,v1:127.0.0.1:6813/3996553280] exists,up 77bdcafa-961d-40b3-b535-4143a4c7ad3b 2024-09-13T14:49:06.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:49:06.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:49:06.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:49:06.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:49:06.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-09-13T14:49:06.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:49:06.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:49:06.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T14:49:06.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:49:06.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T14:49:06.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:49:06.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:49:06.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:49:06.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:49:06.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T14:49:06.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T14:49:06.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:49:06.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:49:06.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:49:06.480 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:49:06.480 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:49:06.480 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:49:06.481 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:49:06.481 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:49:06.481 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:49:06.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:49:06.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:49:06.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:49:06.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:49:06.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:49:06.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:49:06.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:49:06.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:49:06.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:49:06.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:49:06.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:49:06.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:49:06.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:49:06.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T14:49:06.485 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:49:06.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4acd0d33-b2bd-408c-a7d7-17ec2297469e 2024-09-13T14:49:06.485 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 4acd0d33-b2bd-408c-a7d7-17ec2297469e 2024-09-13T14:49:06.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 4acd0d33-b2bd-408c-a7d7-17ec2297469e' 2024-09-13T14:49:06.486 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:49:06.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDiUORmMZzMHRAAKgmmLSTf/jnym7vO9NUiow== 2024-09-13T14:49:06.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDiUORmMZzMHRAAKgmmLSTf/jnym7vO9NUiow=="}' 2024-09-13T14:49:06.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4acd0d33-b2bd-408c-a7d7-17ec2297469e -i td/osd-scrub-repair/2/new.json 2024-09-13T14:49:06.914 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:49:06.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T14:49:06.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDiUORmMZzMHRAAKgmmLSTf/jnym7vO9NUiow== --osd-uuid 4acd0d33-b2bd-408c-a7d7-17ec2297469e 2024-09-13T14:49:06.967 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:06.965+0000 7f180bbbf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:49:06.991 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:06.989+0000 7f180bbbf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:49:06.992 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:06.989+0000 7f180bbbf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:49:06.993 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:06.989+0000 7f180bbbf7c0 -1 bdev(0x563610af4800 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T14:49:06.993 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:06.989+0000 7f180bbbf7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T14:49:12.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T14:49:12.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:49:12.859 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T14:49:12.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T14:49:12.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:49:13.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T14:49:13.256 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T14:49:13.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:49:13.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:49:13.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:49:13.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:49:13.288 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:13.285+0000 7f8e4e7927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:49:13.322 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:13.317+0000 7f8e4e7927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:49:13.323 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:13.321+0000 7f8e4e7927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:49:13.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T14:49:13.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:49:13.590 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:49:13.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T14:49:13.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:49:13.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:49:13.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:13.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:49:13.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:13.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:13.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:14.841 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:14.837+0000 7f8e4e7927c0 -1 Falling back to public interface 2024-09-13T14:49:14.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:14.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:14.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:49:14.913 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:49:14.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:14.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:15.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:16.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:16.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:16.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:49:16.235 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:49:16.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:16.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:16.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:17.547 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:17.545+0000 7f8e4e7927c0 -1 osd.2 0 log_to_monitors true 2024-09-13T14:49:17.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:17.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:17.553 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:49:17.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:49:17.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:17.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:18.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:19.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:19.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:19.042 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:49:19.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:49:19.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:19.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:19.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:20.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:20.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:20.372 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:49:20.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:49:20.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:20.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:20.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:21.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:21.789 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:49:21.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:21.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:49:21.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:21.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:22.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:23.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:23.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:23.160 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:49:23.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:49:23.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:23.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:23.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:24.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:24.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:24.560 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:49:24.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:49:24.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:24.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:25.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:26.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:26.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:26.008 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:49:26.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:49:26.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:26.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:26.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:27.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:27.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:27.446 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:49:27.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:49:27.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:27.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:27.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:28.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:28.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:28.783 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:49:28.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:49:28.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:28.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:29.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:30.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:30.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:30.351 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:49:30.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:49:30.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:30.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:31.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:32.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:32.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:32.037 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:49:32.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:49:32.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:32.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:32.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:33.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:33.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:33.504 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:49:33.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:49:33.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:33.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:34.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:35.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:35.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:35.315 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:49:35.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:49:35.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:35.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:36.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:37.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:37.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:37.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:49:37.154 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:49:37.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:37.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:37.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:38.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:38.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:38.944 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:49:38.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:49:38.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:38.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:39.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:40.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:40.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:40.582 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:49:40.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:49:40.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:40.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T14:49:40.952 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/971535395,v1:127.0.0.1:6819/971535395] [v2:127.0.0.1:6820/971535395,v1:127.0.0.1:6821/971535395] exists,up 4acd0d33-b2bd-408c-a7d7-17ec2297469e 2024-09-13T14:49:40.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:49:40.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:49:40.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:49:40.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:49:40.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-09-13T14:49:40.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:49:40.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:49:40.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-13T14:49:40.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:49:40.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:49:40.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:49:40.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:49:40.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:49:40.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:49:40.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:49:40.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:49:40.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:49:40.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:49:40.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:49:40.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:49:40.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:49:40.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:49:40.958 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:49:40.958 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:49:40.958 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:49:40.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:49:40.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:49:40.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:49:40.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:49:40.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:49:40.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:49:40.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:49:40.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:49:40.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:49:40.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:49:40.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:49:40.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:49:40.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:49:40.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:49:40.960 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:49:40.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=858e19ad-bbe8-4eeb-8da5-f6f455b0cb32 2024-09-13T14:49:40.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 858e19ad-bbe8-4eeb-8da5-f6f455b0cb32' 2024-09-13T14:49:40.960 INFO:tasks.workunit.client.0.smithi112.stdout:add osd3 858e19ad-bbe8-4eeb-8da5-f6f455b0cb32 2024-09-13T14:49:40.960 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:49:40.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAEUeRm9xOJOhAA4lX60gzo5rmki221DH8OLA== 2024-09-13T14:49:40.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAEUeRm9xOJOhAA4lX60gzo5rmki221DH8OLA=="}' 2024-09-13T14:49:40.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 858e19ad-bbe8-4eeb-8da5-f6f455b0cb32 -i td/osd-scrub-repair/3/new.json 2024-09-13T14:49:41.349 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:49:41.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-13T14:49:41.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAEUeRm9xOJOhAA4lX60gzo5rmki221DH8OLA== --osd-uuid 858e19ad-bbe8-4eeb-8da5-f6f455b0cb32 2024-09-13T14:49:41.394 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:41.392+0000 7fd2ff40f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:49:41.434 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:41.432+0000 7fd2ff40f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:49:41.435 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:41.436+0000 7fd2ff40f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:49:41.436 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:41.436+0000 7fd2ff40f7c0 -1 bdev(0x55b3da951c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-13T14:49:41.436 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:41.436+0000 7fd2ff40f7c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-13T14:49:47.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-13T14:49:47.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:49:47.662 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd3 key to auth repository 2024-09-13T14:49:47.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-13T14:49:47.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:49:48.072 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.3 2024-09-13T14:49:48.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-13T14:49:48.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:49:48.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:49:48.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:49:48.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:49:48.105 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:48.104+0000 7f01493397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:49:48.133 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:48.132+0000 7f01493397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:49:48.134 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:48.136+0000 7f01493397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:49:48.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-13T14:49:48.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:49:48.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:49:48.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:49:48.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:49:48.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:48.399 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:49:48.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:49:48.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:49:48.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:48.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:49.673 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:49.672+0000 7f01493397c0 -1 Falling back to public interface 2024-09-13T14:49:49.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:49.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:49.742 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:49:49.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:49:49.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:49.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:49:50.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:51.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:51.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:51.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:49:51.082 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:49:51.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:49:51.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:51.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:52.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:52.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:52.418 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:49:52.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:49:52.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:52.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:49:52.728 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:49:52.728+0000 7f01493397c0 -1 osd.3 0 log_to_monitors true 2024-09-13T14:49:52.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:53.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:53.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:53.760 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:49:53.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:49:53.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:53.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:49:54.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:55.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:55.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:55.098 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:49:55.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:49:55.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:55.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:49:55.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:56.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:56.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:56.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:49:56.575 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:49:56.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:56.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:49:56.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:57.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:57.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:57.935 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:49:57.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:49:57.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:57.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:49:58.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:49:59.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:49:59.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:49:59.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:49:59.272 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:49:59.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:49:59.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:49:59.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:00.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:00.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:00.608 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:50:00.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:50:00.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:00.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:50:00.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:01.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:01.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:01.943 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:50:01.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:50:01.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:01.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:50:02.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:03.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:03.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:03.264 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:50:03.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:50:03.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:03.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:50:03.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:04.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:04.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:04.620 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:50:04.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:50:04.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:04.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:50:04.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:05.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:05.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:05.959 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:50:05.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:50:05.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:05.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:50:06.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:07.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:07.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:07.291 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:50:07.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:50:07.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:07.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:50:07.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:08.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:08.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:08.995 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:50:08.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:50:08.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:08.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:50:09.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:10.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:10.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:10.760 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:50:10.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:50:10.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:10.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:50:11.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:12.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:12.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:12.688 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:50:12.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:50:12.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:12.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:50:13.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:14.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:14.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:14.148 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:50:14.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:50:14.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:14.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:50:14.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:15.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:15.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:15.532 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:50:15.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:50:15.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:15.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:50:16.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:17.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:17.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:17.284 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T14:50:17.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T14:50:17.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:17.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:50:17.668 INFO:tasks.workunit.client.0.smithi112.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4105137706,v1:127.0.0.1:6827/4105137706] [v2:127.0.0.1:6828/4105137706,v1:127.0.0.1:6829/4105137706] exists,up 858e19ad-bbe8-4eeb-8da5-f6f455b0cb32 2024-09-13T14:50:17.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:50:17.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:50:17.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:50:17.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:50:17.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-09-13T14:50:17.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:50:17.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:50:17.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-13T14:50:17.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:50:17.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/4 2024-09-13T14:50:17.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:50:17.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:50:17.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:50:17.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:50:17.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-09-13T14:50:17.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-09-13T14:50:17.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:50:17.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:50:17.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:50:17.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:50:17.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:50:17.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:50:17.673 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:50:17.673 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:50:17.673 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:50:17.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:50:17.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:50:17.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:50:17.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:50:17.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:50:17.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:50:17.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:50:17.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:50:17.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:50:17.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:50:17.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:50:17.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:50:17.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:50:17.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/4 2024-09-13T14:50:17.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:50:17.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=be73f37e-bde5-4626-bab0-80f27335126d 2024-09-13T14:50:17.675 INFO:tasks.workunit.client.0.smithi112.stdout:add osd4 be73f37e-bde5-4626-bab0-80f27335126d 2024-09-13T14:50:17.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 be73f37e-bde5-4626-bab0-80f27335126d' 2024-09-13T14:50:17.675 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:50:17.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQApUeRmWp+eKRAA2l2qiICvbRmJ7V1xvV+zpg== 2024-09-13T14:50:17.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQApUeRmWp+eKRAA2l2qiICvbRmJ7V1xvV+zpg=="}' 2024-09-13T14:50:17.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new be73f37e-bde5-4626-bab0-80f27335126d -i td/osd-scrub-repair/4/new.json 2024-09-13T14:50:18.102 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:50:18.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/4/new.json 2024-09-13T14:50:18.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQApUeRmWp+eKRAA2l2qiICvbRmJ7V1xvV+zpg== --osd-uuid be73f37e-bde5-4626-bab0-80f27335126d 2024-09-13T14:50:18.161 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:18.159+0000 7f7c42ce37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:50:18.187 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:18.187+0000 7f7c42ce37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:50:18.189 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:18.187+0000 7f7c42ce37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:50:18.189 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:18.187+0000 7f7c42ce37c0 -1 bdev(0x557d2b2ebc00 td/osd-scrub-repair/4/block) open stat got: (1) Operation not permitted 2024-09-13T14:50:18.189 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:18.187+0000 7f7c42ce37c0 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-09-13T14:50:24.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-09-13T14:50:24.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:50:24.612 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd4 key to auth repository 2024-09-13T14:50:24.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-13T14:50:24.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:50:25.010 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.4 2024-09-13T14:50:25.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-13T14:50:25.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:50:25.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:50:25.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:50:25.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:50:25.043 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:25.043+0000 7f0101cbf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:50:25.045 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:25.043+0000 7f0101cbf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:50:25.046 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:25.043+0000 7f0101cbf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:50:25.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-13T14:50:25.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:50:25.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-13T14:50:25.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:50:25.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:50:25.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:25.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:50:25.364 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:50:25.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:25.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:25.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:26.357 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:26.355+0000 7f0101cbf7c0 -1 Falling back to public interface 2024-09-13T14:50:26.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:26.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:26.702 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:50:26.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:50:26.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:26.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:27.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:28.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:28.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:28.041 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:50:28.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:50:28.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:28.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:28.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:29.117 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:29.115+0000 7f0101cbf7c0 -1 osd.4 0 log_to_monitors true 2024-09-13T14:50:29.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:29.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:29.369 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:50:29.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:50:29.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:29.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:29.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:30.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:30.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:30.758 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:50:30.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:50:30.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:30.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:31.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:32.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:32.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:32.092 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:50:32.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:50:32.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:32.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:32.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:33.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:33.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:33.424 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:50:33.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:50:33.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:33.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:33.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:34.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:34.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:34.850 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:50:34.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:50:34.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:34.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:35.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:36.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:36.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:36.287 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:50:36.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:50:36.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:36.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:36.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:37.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:37.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:37.621 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:50:37.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:50:37.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:37.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:38.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:39.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:39.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:39.044 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:50:39.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:50:39.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:39.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:39.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:40.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:40.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:40.367 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:50:40.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:50:40.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:40.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:40.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:41.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:41.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:41.937 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:50:41.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:50:41.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:41.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:42.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:43.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:43.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:43.608 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:50:43.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:50:43.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:43.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:44.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:45.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:45.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:45.431 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:50:45.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:50:45.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:45.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:45.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:47.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:47.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:47.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:50:47.002 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:50:47.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:47.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:47.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:48.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:48.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:48.401 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:50:48.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:50:48.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:48.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:48.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:49.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:49.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:49.794 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:50:49.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:50:49.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:49.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:50.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:50:51.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:50:51.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:51.258 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:50:51.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:50:51.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:51.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-13T14:50:51.673 INFO:tasks.workunit.client.0.smithi112.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3050797907,v1:127.0.0.1:6835/3050797907] [v2:127.0.0.1:6836/3050797907,v1:127.0.0.1:6837/3050797907] exists,up be73f37e-bde5-4626-bab0-80f27335126d 2024-09-13T14:50:51.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:50:51.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:50:51.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:50:51.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:50:51.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-09-13T14:50:51.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:50:51.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:50:51.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-13T14:50:51.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:50:51.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:50:51.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:50:51.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:50:51.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:50:51.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:50:51.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-13T14:50:51.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-13T14:50:51.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:50:51.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:50:51.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:50:51.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:50:51.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:50:51.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:50:51.679 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:50:51.679 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:50:51.679 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:50:51.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:50:51.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:50:51.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:50:51.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:50:51.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:50:51.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:50:51.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:50:51.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:50:51.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:50:51.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:50:51.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:50:51.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:50:51.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:50:51.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/5 2024-09-13T14:50:51.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:50:51.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3df67c99-00a1-4d12-b468-6f9b9faac1b1 2024-09-13T14:50:51.711 INFO:tasks.workunit.client.0.smithi112.stdout:add osd5 3df67c99-00a1-4d12-b468-6f9b9faac1b1 2024-09-13T14:50:51.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 3df67c99-00a1-4d12-b468-6f9b9faac1b1' 2024-09-13T14:50:51.712 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:50:51.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBLUeRmwnuYKxAAVHwh15eqDZlNF+Z6Dk3Vnw== 2024-09-13T14:50:51.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBLUeRmwnuYKxAAVHwh15eqDZlNF+Z6Dk3Vnw=="}' 2024-09-13T14:50:51.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3df67c99-00a1-4d12-b468-6f9b9faac1b1 -i td/osd-scrub-repair/5/new.json 2024-09-13T14:50:52.090 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:50:52.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/5/new.json 2024-09-13T14:50:52.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLUeRmwnuYKxAAVHwh15eqDZlNF+Z6Dk3Vnw== --osd-uuid 3df67c99-00a1-4d12-b468-6f9b9faac1b1 2024-09-13T14:50:52.145 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:52.142+0000 7fcec10097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:50:52.186 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:52.186+0000 7fcec10097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:50:52.188 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:52.186+0000 7fcec10097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:50:52.188 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:52.186+0000 7fcec10097c0 -1 bdev(0x56025ae15c00 td/osd-scrub-repair/5/block) open stat got: (1) Operation not permitted 2024-09-13T14:50:52.188 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:52.186+0000 7fcec10097c0 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-09-13T14:50:58.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-09-13T14:50:58.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:50:58.336 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd5 key to auth repository 2024-09-13T14:50:58.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-13T14:50:58.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:50:58.752 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.5 2024-09-13T14:50:58.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-13T14:50:58.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:50:58.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:50:58.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:50:58.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:50:58.785 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:58.782+0000 7fd507eda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:50:58.807 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:58.806+0000 7fd507eda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:50:58.808 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:50:58.806+0000 7fd507eda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:50:59.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-13T14:50:59.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:50:59.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-13T14:50:59.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:50:59.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:50:59.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:50:59.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:50:59.079 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:50:59.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:50:59.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:50:59.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:00.381 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:51:00.378+0000 7fd507eda7c0 -1 Falling back to public interface 2024-09-13T14:51:00.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:00.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:00.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:51:00.439 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:51:00.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:00.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:00.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:01.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:01.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:01.775 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:51:01.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:51:01.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:01.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:02.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:03.090 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:51:03.090+0000 7fd507eda7c0 -1 osd.5 0 log_to_monitors true 2024-09-13T14:51:03.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:03.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:03.098 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:51:03.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:51:03.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:03.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:03.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:04.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:04.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:04.727 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:51:04.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:51:04.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:04.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:05.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:06.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:06.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:06.059 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:51:06.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:51:06.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:06.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:06.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:07.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:07.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:07.409 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:51:07.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:51:07.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:07.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:07.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:08.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:08.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:08.742 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:51:08.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:51:08.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:08.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:09.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:10.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:10.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:10.150 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:51:10.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:51:10.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:10.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:10.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:11.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:11.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:11.477 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:51:11.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:51:11.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:11.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:11.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:12.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:12.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:12.843 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:51:12.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:51:12.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:12.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:13.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:14.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:14.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:14.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:51:14.310 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:51:14.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:14.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:14.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:15.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:15.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:15.798 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:51:15.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:51:15.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:15.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:16.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:17.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:17.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:17.143 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:51:17.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:51:17.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:17.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:17.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:18.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:18.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:18.647 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:51:18.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:51:18.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:18.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:19.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:20.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:20.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:20.502 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:51:20.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:51:20.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:20.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:21.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:22.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:22.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:22.236 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:51:22.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:51:22.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:22.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:22.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:23.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:23.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:23.633 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:51:23.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:51:23.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:23.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:24.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:25.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:25.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:25.216 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:51:25.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:51:25.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:25.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:25.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:26.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:26.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:26.760 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:51:26.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:51:26.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:26.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:51:27.091 INFO:tasks.workunit.client.0.smithi112.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3688619572,v1:127.0.0.1:6843/3688619572] [v2:127.0.0.1:6844/3688619572,v1:127.0.0.1:6845/3688619572] exists,up 3df67c99-00a1-4d12-b468-6f9b9faac1b1 2024-09-13T14:51:27.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:51:27.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:51:27.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:51:27.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:51:27.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-09-13T14:51:27.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:51:27.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:51:27.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-13T14:51:27.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:51:27.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/6 2024-09-13T14:51:27.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:51:27.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:51:27.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:51:27.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:51:27.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-09-13T14:51:27.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-09-13T14:51:27.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:51:27.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:51:27.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:51:27.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:51:27.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:51:27.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:51:27.097 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:51:27.097 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:51:27.097 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:51:27.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:51:27.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:51:27.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:51:27.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:51:27.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:51:27.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:51:27.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:51:27.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:51:27.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:51:27.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:51:27.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:51:27.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:51:27.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:51:27.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/6 2024-09-13T14:51:27.099 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:51:27.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a70a15a7-3b82-4958-beae-295641f8c567 2024-09-13T14:51:27.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 a70a15a7-3b82-4958-beae-295641f8c567' 2024-09-13T14:51:27.099 INFO:tasks.workunit.client.0.smithi112.stdout:add osd6 a70a15a7-3b82-4958-beae-295641f8c567 2024-09-13T14:51:27.099 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:51:27.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBvUeRmOCxSBxAAazqQyTVRFHN6+Ph7X/6Iag== 2024-09-13T14:51:27.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBvUeRmOCxSBxAAazqQyTVRFHN6+Ph7X/6Iag=="}' 2024-09-13T14:51:27.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a70a15a7-3b82-4958-beae-295641f8c567 -i td/osd-scrub-repair/6/new.json 2024-09-13T14:51:27.460 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:51:27.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/6/new.json 2024-09-13T14:51:27.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvUeRmOCxSBxAAazqQyTVRFHN6+Ph7X/6Iag== --osd-uuid a70a15a7-3b82-4958-beae-295641f8c567 2024-09-13T14:51:27.513 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:51:27.510+0000 7f452644a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:51:27.515 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:51:27.514+0000 7f452644a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:51:27.516 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:51:27.514+0000 7f452644a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:51:27.516 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:51:27.514+0000 7f452644a7c0 -1 bdev(0x5645d0c9bc00 td/osd-scrub-repair/6/block) open stat got: (1) Operation not permitted 2024-09-13T14:51:27.516 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:51:27.514+0000 7f452644a7c0 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-09-13T14:51:33.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-09-13T14:51:33.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:51:33.894 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd6 key to auth repository 2024-09-13T14:51:33.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-13T14:51:33.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:51:34.306 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.6 2024-09-13T14:51:34.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-13T14:51:34.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:51:34.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:51:34.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:51:34.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:51:34.344 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:51:34.342+0000 7f311bf2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:51:34.360 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:51:34.358+0000 7f311bf2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:51:34.361 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:51:34.358+0000 7f311bf2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:51:34.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-13T14:51:34.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:51:34.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-13T14:51:34.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:51:34.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:51:34.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:34.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:51:34.636 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:51:34.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:34.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:34.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:35.633 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:51:35.629+0000 7f311bf2f7c0 -1 Falling back to public interface 2024-09-13T14:51:35.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:35.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:35.945 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:51:35.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:51:35.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:35.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:36.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:37.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:37.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:37.275 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:51:37.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:51:37.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:37.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:37.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:38.306 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:51:38.305+0000 7f311bf2f7c0 -1 osd.6 0 log_to_monitors true 2024-09-13T14:51:38.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:38.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:38.612 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:51:38.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:51:38.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:38.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:38.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:39.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:39.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:39.973 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:51:39.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:51:39.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:39.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:40.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:41.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:41.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:41.307 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:51:41.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:51:41.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:41.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:41.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:42.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:42.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:42.634 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:51:42.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:51:42.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:42.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:43.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:44.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:44.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:44.157 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:51:44.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:51:44.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:44.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:44.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:45.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:45.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:45.495 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:51:45.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:51:45.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:45.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:45.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:46.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:46.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:46.903 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:51:46.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:51:46.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:46.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:47.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:48.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:48.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:48.351 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:51:48.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:51:48.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:48.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:48.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:49.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:49.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:49.680 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:51:49.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:51:49.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:49.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:50.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:51.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:51.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:51.054 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:51:51.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:51:51.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:51.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:51.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:52.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:52.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:52.711 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:51:52.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:51:52.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:52.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:53.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:54.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:54.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:54.752 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:51:54.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:51:54.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:54.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:55.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:56.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:56.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:56.432 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:51:56.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:51:56.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:56.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:56.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:57.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:57.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:57.907 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:51:57.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:51:57.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:57.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:51:58.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:51:59.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:51:59.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:51:59.703 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:51:59.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:51:59.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:51:59.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:52:00.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:01.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:01.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:01.584 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:52:01.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:52:01.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:01.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-13T14:52:01.976 INFO:tasks.workunit.client.0.smithi112.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2360308477,v1:127.0.0.1:6851/2360308477] [v2:127.0.0.1:6852/2360308477,v1:127.0.0.1:6853/2360308477] exists,up a70a15a7-3b82-4958-beae-295641f8c567 2024-09-13T14:52:01.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:52:01.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:52:01.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:52:01.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:52:01.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-09-13T14:52:01.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:52:01.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:52:01.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-13T14:52:01.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:52:01.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/7 2024-09-13T14:52:01.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:52:01.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:52:01.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:52:01.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:52:01.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-09-13T14:52:01.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-09-13T14:52:01.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:52:01.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:52:01.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:52:01.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:52:01.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:52:01.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:52:01.984 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:52:01.984 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:52:01.984 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:52:01.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:52:01.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:52:01.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:52:01.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:52:01.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:52:01.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:52:01.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:52:01.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:52:01.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:52:01.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:52:01.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:52:01.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:52:01.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:52:01.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/7 2024-09-13T14:52:01.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:52:01.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=228ff212-100a-4b72-9d1b-60522b255a77 2024-09-13T14:52:01.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 228ff212-100a-4b72-9d1b-60522b255a77' 2024-09-13T14:52:01.987 INFO:tasks.workunit.client.0.smithi112.stdout:add osd7 228ff212-100a-4b72-9d1b-60522b255a77 2024-09-13T14:52:01.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:52:02.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCSUeRmg9ddABAAhpGrtqvqWM1psns1CTm7RQ== 2024-09-13T14:52:02.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCSUeRmg9ddABAAhpGrtqvqWM1psns1CTm7RQ=="}' 2024-09-13T14:52:02.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 228ff212-100a-4b72-9d1b-60522b255a77 -i td/osd-scrub-repair/7/new.json 2024-09-13T14:52:02.464 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:52:02.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/7/new.json 2024-09-13T14:52:02.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCSUeRmg9ddABAAhpGrtqvqWM1psns1CTm7RQ== --osd-uuid 228ff212-100a-4b72-9d1b-60522b255a77 2024-09-13T14:52:02.524 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:02.521+0000 7f7d7fe007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:52:02.534 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:02.533+0000 7f7d7fe007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:52:02.535 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:02.533+0000 7f7d7fe007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:52:02.536 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:02.533+0000 7f7d7fe007c0 -1 bdev(0x557646ca9c00 td/osd-scrub-repair/7/block) open stat got: (1) Operation not permitted 2024-09-13T14:52:02.536 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:02.533+0000 7f7d7fe007c0 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-09-13T14:52:09.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-09-13T14:52:09.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:52:09.284 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd7 key to auth repository 2024-09-13T14:52:09.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-13T14:52:09.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:52:09.694 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.7 2024-09-13T14:52:09.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-13T14:52:09.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:52:09.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:52:09.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:52:09.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:52:09.728 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:09.725+0000 7ff4ac0107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:52:09.748 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:09.745+0000 7ff4ac0107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:52:09.749 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:09.745+0000 7ff4ac0107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:52:10.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-13T14:52:10.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:52:10.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-13T14:52:10.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:52:10.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:52:10.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:10.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:52:10.031 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:52:10.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:10.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:10.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:11.020 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:11.017+0000 7ff4ac0107c0 -1 Falling back to public interface 2024-09-13T14:52:11.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:11.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:11.398 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:52:11.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:52:11.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:11.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:11.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:12.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:12.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:12.737 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:52:12.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:52:12.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:12.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:13.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:13.688 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:13.685+0000 7ff4ac0107c0 -1 osd.7 0 log_to_monitors true 2024-09-13T14:52:14.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:14.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:14.066 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:52:14.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:52:14.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:14.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:14.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:15.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:15.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:15.506 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:52:15.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:52:15.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:15.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:15.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:16.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:16.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:16.838 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:52:16.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:52:16.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:16.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:17.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:18.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:18.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:18.187 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:52:18.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:52:18.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:18.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:18.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:19.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:19.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:19.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:52:19.589 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:52:19.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:19.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:19.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:20.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:20.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:20.917 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:52:20.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:52:20.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:20.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:21.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:22.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:22.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:22.256 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:52:22.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:52:22.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:22.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:22.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:23.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:23.590 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:52:23.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:52:23.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:23.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:23.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:24.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:24.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:24.932 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:52:24.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:52:24.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:24.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:25.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:26.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:26.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:26.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:52:26.265 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:52:26.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:26.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:26.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:27.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:27.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:27.778 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:52:27.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:52:27.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:27.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:28.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:29.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:29.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:29.245 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:52:29.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:52:29.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:29.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:29.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:30.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:30.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:30.799 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:52:30.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:52:30.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:30.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:31.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:32.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:32.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:32.560 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:52:32.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:52:32.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:32.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:33.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:34.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:34.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:34.137 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:52:34.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:52:34.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:34.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:34.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:35.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:35.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:35.721 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:52:35.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:52:35.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:35.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:36.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:37.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:37.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:37.271 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:52:37.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:52:37.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:37.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-13T14:52:37.608 INFO:tasks.workunit.client.0.smithi112.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1488565940,v1:127.0.0.1:6859/1488565940] [v2:127.0.0.1:6860/1488565940,v1:127.0.0.1:6861/1488565940] exists,up 228ff212-100a-4b72-9d1b-60522b255a77 2024-09-13T14:52:37.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:52:37.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:52:37.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:52:37.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:52:37.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-09-13T14:52:37.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:52:37.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:52:37.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2024-09-13T14:52:37.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:52:37.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/8 2024-09-13T14:52:37.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:52:37.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:52:37.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:52:37.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:52:37.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-09-13T14:52:37.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-09-13T14:52:37.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:52:37.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:52:37.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:52:37.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:52:37.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:52:37.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:52:37.612 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:52:37.612 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:52:37.612 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:52:37.613 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:52:37.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:52:37.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:52:37.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:52:37.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:52:37.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:52:37.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:52:37.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:52:37.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:52:37.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:52:37.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:52:37.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:52:37.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:52:37.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/8 2024-09-13T14:52:37.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:52:37.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b866cc0e-43dd-449f-9c80-d29ba519c905 2024-09-13T14:52:37.614 INFO:tasks.workunit.client.0.smithi112.stdout:add osd8 b866cc0e-43dd-449f-9c80-d29ba519c905 2024-09-13T14:52:37.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 b866cc0e-43dd-449f-9c80-d29ba519c905' 2024-09-13T14:52:37.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:52:37.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC1UeRmNDYOJhAAjvc71ZZWeSWHvph/hyahpQ== 2024-09-13T14:52:37.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC1UeRmNDYOJhAAjvc71ZZWeSWHvph/hyahpQ=="}' 2024-09-13T14:52:37.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b866cc0e-43dd-449f-9c80-d29ba519c905 -i td/osd-scrub-repair/8/new.json 2024-09-13T14:52:38.057 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:52:38.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/8/new.json 2024-09-13T14:52:38.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1UeRmNDYOJhAAjvc71ZZWeSWHvph/hyahpQ== --osd-uuid b866cc0e-43dd-449f-9c80-d29ba519c905 2024-09-13T14:52:38.114 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:38.116+0000 7f274d23e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:52:38.116 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:38.116+0000 7f274d23e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:52:38.118 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:38.120+0000 7f274d23e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:52:38.118 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:38.120+0000 7f274d23e7c0 -1 bdev(0x55b42d0d7c00 td/osd-scrub-repair/8/block) open stat got: (1) Operation not permitted 2024-09-13T14:52:38.118 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:38.120+0000 7f274d23e7c0 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-09-13T14:52:44.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-09-13T14:52:44.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:52:44.498 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd8 key to auth repository 2024-09-13T14:52:44.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2024-09-13T14:52:44.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:52:44.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2024-09-13T14:52:44.901 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.8 2024-09-13T14:52:44.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:52:44.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:52:44.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:52:44.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:52:44.933 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:44.932+0000 7fee788f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:52:44.935 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:44.936+0000 7fee788f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:52:44.937 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:44.936+0000 7fee788f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:52:45.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2024-09-13T14:52:45.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:52:45.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2024-09-13T14:52:45.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:52:45.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:52:45.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:45.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:52:45.244 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:52:45.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:45.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:52:45.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:46.228 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:46.228+0000 7fee788f67c0 -1 Falling back to public interface 2024-09-13T14:52:46.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:46.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:46.609 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:52:46.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:52:46.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:46.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:52:46.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:47.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:47.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:47.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:52:47.943 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:52:47.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:47.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:52:48.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:49.157 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:52:49.156+0000 7fee788f67c0 -1 osd.8 0 log_to_monitors true 2024-09-13T14:52:49.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:49.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:49.264 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:52:49.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:52:49.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:49.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:52:49.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:50.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:50.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:50.673 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:52:50.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:52:50.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:50.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:52:51.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:52.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:52.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:52.023 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:52:52.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:52:52.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:52.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:52:52.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:53.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:53.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:53.407 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:52:53.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:52:53.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:53.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:52:53.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:54.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:54.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:54.759 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:52:54.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:52:54.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:54.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:52:55.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:56.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:56.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:56.097 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:52:56.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:52:56.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:56.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:52:56.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:57.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:57.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:57.439 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:52:57.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:52:57.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:57.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:52:57.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:52:58.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:52:58.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:52:58.830 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:52:58.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:52:58.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:52:58.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:52:59.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:00.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:00.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:00.168 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:53:00.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:53:00.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:00.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:53:00.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:01.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:01.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:01.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:53:01.508 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:53:01.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:01.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:53:01.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:02.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:02.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:02.846 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:53:02.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:53:02.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:02.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:53:03.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:04.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:04.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:04.573 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:53:04.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:53:04.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:04.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:53:05.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:06.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:06.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:06.406 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:53:06.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:53:06.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:06.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:53:07.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:08.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:08.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:08.362 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:53:08.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:53:08.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:08.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:53:09.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:10.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:10.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:10.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:53:10.154 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:53:10.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:10.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:53:10.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:11.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:11.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:11.492 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:53:11.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:53:11.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:11.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:53:12.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:13.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:13.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:13.021 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:53:13.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:53:13.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:13.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-13T14:53:13.353 INFO:tasks.workunit.client.0.smithi112.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/3932055523,v1:127.0.0.1:6867/3932055523] [v2:127.0.0.1:6868/3932055523,v1:127.0.0.1:6869/3932055523] exists,up b866cc0e-43dd-449f-9c80-d29ba519c905 2024-09-13T14:53:13.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:53:13.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:53:13.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:53:13.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-13T14:53:13.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-09-13T14:53:13.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:53:13.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:53:13.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2024-09-13T14:53:13.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:53:13.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:53:13.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:53:13.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:53:13.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:53:13.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:53:13.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-13T14:53:13.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-13T14:53:13.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:53:13.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:53:13.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:53:13.365 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:53:13.365 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:53:13.365 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:53:13.365 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:53:13.365 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:53:13.365 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:53:13.366 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:53:13.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:53:13.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:53:13.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:53:13.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:53:13.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:53:13.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:53:13.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:53:13.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:53:13.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:53:13.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:53:13.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:53:13.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:53:13.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/9 2024-09-13T14:53:13.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:53:13.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0253c251-b81a-4f43-af92-c81fb0549c27 2024-09-13T14:53:13.378 INFO:tasks.workunit.client.0.smithi112.stdout:add osd9 0253c251-b81a-4f43-af92-c81fb0549c27 2024-09-13T14:53:13.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 0253c251-b81a-4f43-af92-c81fb0549c27' 2024-09-13T14:53:13.379 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:53:13.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDZUeRmJ/O5FxAA2RrgzqHdWO3Fg3iO4zLhBQ== 2024-09-13T14:53:13.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDZUeRmJ/O5FxAA2RrgzqHdWO3Fg3iO4zLhBQ=="}' 2024-09-13T14:53:13.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0253c251-b81a-4f43-af92-c81fb0549c27 -i td/osd-scrub-repair/9/new.json 2024-09-13T14:53:13.826 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:53:13.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/9/new.json 2024-09-13T14:53:13.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZUeRmJ/O5FxAA2RrgzqHdWO3Fg3iO4zLhBQ== --osd-uuid 0253c251-b81a-4f43-af92-c81fb0549c27 2024-09-13T14:53:13.881 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:53:13.879+0000 7f5d262317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:53:13.903 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:53:13.903+0000 7f5d262317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:53:13.904 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:53:13.903+0000 7f5d262317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:53:13.904 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:53:13.903+0000 7f5d262317c0 -1 bdev(0x563c35cc3c00 td/osd-scrub-repair/9/block) open stat got: (1) Operation not permitted 2024-09-13T14:53:13.904 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:53:13.903+0000 7f5d262317c0 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-09-13T14:53:20.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-09-13T14:53:20.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:53:20.549 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd9 key to auth repository 2024-09-13T14:53:20.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2024-09-13T14:53:20.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:53:20.956 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.9 2024-09-13T14:53:20.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2024-09-13T14:53:20.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:53:20.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:53:20.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:53:20.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:53:20.988 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:53:20.987+0000 7fec577cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:53:21.014 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:53:21.015+0000 7fec577cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:53:21.015 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:53:21.015+0000 7fec577cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:53:21.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2024-09-13T14:53:21.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:53:21.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-13T14:53:21.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:53:21.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:53:21.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:21.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:53:21.280 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:53:21.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:21.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:21.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:22.097 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:53:22.095+0000 7fec577cc7c0 -1 Falling back to public interface 2024-09-13T14:53:22.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:22.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:22.612 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:53:22.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:53:22.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:22.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:22.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:23.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:23.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:23.938 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:53:23.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:53:23.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:23.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:24.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:24.963 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:53:24.963+0000 7fec577cc7c0 -1 osd.9 0 log_to_monitors true 2024-09-13T14:53:25.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:25.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:25.256 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:53:25.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:53:25.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:25.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:25.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:26.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:26.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:26.635 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:53:26.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:53:26.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:26.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:27.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:28.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:28.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:28.122 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:53:28.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:53:28.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:28.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:28.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:29.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:29.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:29.463 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:53:29.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:53:29.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:29.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:29.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:30.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:30.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:30.794 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:53:30.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:53:30.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:30.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:31.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:32.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:32.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:32.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:53:32.218 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:53:32.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:32.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:32.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:33.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:33.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:33.555 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:53:33.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:53:33.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:33.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:33.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:34.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:34.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:34.930 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:53:34.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:53:34.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:34.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:35.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:36.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:36.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:36.272 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:53:36.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:53:36.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:36.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:36.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:37.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:37.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:37.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:53:37.630 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:53:37.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:37.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:38.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:39.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:39.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:39.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:53:39.048 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:53:39.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:39.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:39.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:40.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:40.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:40.480 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:53:40.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:53:40.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:40.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:41.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:42.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:42.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:42.215 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:53:42.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:53:42.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:42.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:42.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:43.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:43.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:43.758 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:53:43.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:53:43.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:43.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:44.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:45.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:45.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:45.434 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:53:45.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:53:45.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:45.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:45.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:46.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:46.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:46.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:53:46.927 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:53:46.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:46.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:47.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:48.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:48.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:48.485 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:53:48.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:53:48.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:48.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:48.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:53:49.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:53:49.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:53:49.926 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T14:53:49.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T14:53:49.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:53:49.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:53:50.288 INFO:tasks.workunit.client.0.smithi112.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2010830029,v1:127.0.0.1:6875/2010830029] [v2:127.0.0.1:6876/2010830029,v1:127.0.0.1:6877/2010830029] exists,up 0253c251-b81a-4f43-af92-c81fb0549c27 2024-09-13T14:53:50.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:53:50.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:53:50.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:53:50.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-09-13T14:53:50.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T14:53:50.668 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' does not exist 2024-09-13T14:53:50.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T14:53:50.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T14:53:51.166 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T14:53:51.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:53:52.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T14:53:54.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-09-13T14:53:54.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:53:54.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:53:54.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:53:54.965 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:53:54.966 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:53:54.966 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:53:54.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:53:54.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:53:54.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:53:55.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:53:55.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:53:55.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:53:55.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:53:55.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:53:55.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:53:55.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:53:55.565 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:53:55.565 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:53:55.565 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:53:55.565 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:53:55.565 INFO:tasks.workunit.client.0.smithi112.stderr:5 2024-09-13T14:53:55.565 INFO:tasks.workunit.client.0.smithi112.stderr:6 2024-09-13T14:53:55.565 INFO:tasks.workunit.client.0.smithi112.stderr:7 2024-09-13T14:53:55.565 INFO:tasks.workunit.client.0.smithi112.stderr:8 2024-09-13T14:53:55.566 INFO:tasks.workunit.client.0.smithi112.stderr:9' 2024-09-13T14:53:55.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:53:55.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:53:55.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:53:55.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836550 2024-09-13T14:53:55.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836550 2024-09-13T14:53:55.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550' 2024-09-13T14:53:55.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:53:55.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:53:55.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673023 2024-09-13T14:53:55.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673023 2024-09-13T14:53:55.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673023' 2024-09-13T14:53:55.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:53:55.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:53:56.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509495 2024-09-13T14:53:56.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509495 2024-09-13T14:53:56.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673023 2-64424509495' 2024-09-13T14:53:56.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:53:56.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:53:56.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345970 2024-09-13T14:53:56.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345970 2024-09-13T14:53:56.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673023 2-64424509495 3-85899345970' 2024-09-13T14:53:56.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:53:56.167 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:53:56.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182443 2024-09-13T14:53:56.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182443 2024-09-13T14:53:56.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673023 2-64424509495 3-85899345970 4-107374182443' 2024-09-13T14:53:56.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:53:56.312 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:53:56.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018914 2024-09-13T14:53:56.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018914 2024-09-13T14:53:56.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673023 2-64424509495 3-85899345970 4-107374182443 5-128849018914' 2024-09-13T14:53:56.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:53:56.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:53:56.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855388 2024-09-13T14:53:56.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855388 2024-09-13T14:53:56.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673023 2-64424509495 3-85899345970 4-107374182443 5-128849018914 6-150323855388' 2024-09-13T14:53:56.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:53:56.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:53:56.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691860 2024-09-13T14:53:56.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691860 2024-09-13T14:53:56.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673023 2-64424509495 3-85899345970 4-107374182443 5-128849018914 6-150323855388 7-171798691860' 2024-09-13T14:53:56.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:53:56.771 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:53:56.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528337 2024-09-13T14:53:56.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528337 2024-09-13T14:53:56.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673023 2-64424509495 3-85899345970 4-107374182443 5-128849018914 6-150323855388 7-171798691860 8-193273528337' 2024-09-13T14:53:56.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:53:56.903 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:53:57.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364809 2024-09-13T14:53:57.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364809 2024-09-13T14:53:57.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673023 2-64424509495 3-85899345970 4-107374182443 5-128849018914 6-150323855388 7-171798691860 8-193273528337 9-214748364809' 2024-09-13T14:53:57.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:53:57.046 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836550 2024-09-13T14:53:57.046 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:53:57.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:53:57.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:53:57.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836550 2024-09-13T14:53:57.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836550 2024-09-13T14:53:57.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-09-13T14:53:57.049 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836550 2024-09-13T14:53:57.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:53:57.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836551 -lt 21474836550 2024-09-13T14:53:57.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:53:57.365 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673023 2024-09-13T14:53:57.366 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:53:57.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:53:57.367 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673023 2024-09-13T14:53:57.367 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:53:57.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673023 2024-09-13T14:53:57.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2024-09-13T14:53:57.369 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673023 2024-09-13T14:53:57.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:53:57.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673023 -lt 42949673023 2024-09-13T14:53:57.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:53:57.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509495 2024-09-13T14:53:57.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:53:57.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:53:57.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509495 2024-09-13T14:53:57.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:53:57.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509495 2024-09-13T14:53:57.688 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509495 2024-09-13T14:53:57.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509495' 2024-09-13T14:53:57.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:53:58.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509496 -lt 64424509495 2024-09-13T14:53:58.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:53:58.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345970 2024-09-13T14:53:58.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:53:58.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:53:58.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345970 2024-09-13T14:53:58.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:53:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345970 2024-09-13T14:53:58.024 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.3 seq 85899345970 2024-09-13T14:53:58.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345970' 2024-09-13T14:53:58.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:53:58.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345971 -lt 85899345970 2024-09-13T14:53:58.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:53:58.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:53:58.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182443 2024-09-13T14:53:58.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:53:58.349 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182443 2024-09-13T14:53:58.349 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:53:58.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182443 2024-09-13T14:53:58.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182443' 2024-09-13T14:53:58.350 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.4 seq 107374182443 2024-09-13T14:53:58.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:53:58.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182444 -lt 107374182443 2024-09-13T14:53:58.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:53:58.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018914 2024-09-13T14:53:58.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:53:58.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:53:58.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018914 2024-09-13T14:53:58.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:53:58.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018914 2024-09-13T14:53:58.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018914' 2024-09-13T14:53:58.661 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.5 seq 128849018914 2024-09-13T14:53:58.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:53:58.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018915 -lt 128849018914 2024-09-13T14:53:58.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:53:58.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855388 2024-09-13T14:53:58.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:53:58.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:53:58.960 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855388 2024-09-13T14:53:58.960 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:53:58.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855388 2024-09-13T14:53:58.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855388' 2024-09-13T14:53:58.961 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.6 seq 150323855388 2024-09-13T14:53:58.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:53:59.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855389 -lt 150323855388 2024-09-13T14:53:59.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:53:59.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691860 2024-09-13T14:53:59.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:53:59.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:53:59.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:53:59.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691860 2024-09-13T14:53:59.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691860 2024-09-13T14:53:59.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691860' 2024-09-13T14:53:59.353 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.7 seq 171798691860 2024-09-13T14:53:59.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:53:59.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691861 -lt 171798691860 2024-09-13T14:53:59.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:53:59.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528337 2024-09-13T14:53:59.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:53:59.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:53:59.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528337 2024-09-13T14:53:59.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:53:59.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528337 2024-09-13T14:53:59.710 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.8 seq 193273528337 2024-09-13T14:53:59.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528337' 2024-09-13T14:53:59.711 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:54:00.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528337 -lt 193273528337 2024-09-13T14:54:00.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:00.058 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364809 2024-09-13T14:54:00.058 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:00.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:54:00.060 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364809 2024-09-13T14:54:00.060 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:00.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364809 2024-09-13T14:54:00.066 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.9 seq 214748364809 2024-09-13T14:54:00.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364809' 2024-09-13T14:54:00.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:54:00.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364810 -lt 214748364809 2024-09-13T14:54:00.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:54:00.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:54:00.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:54:00.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T14:54:00.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:54:00.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:54:00.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:54:00.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:54:00.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:54:00.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:54:00.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:54:01.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:54:01.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:54:01.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:54:01.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:54:01.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T14:54:01.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:54:01.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:54:01.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-09-13T14:54:01.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-13T14:54:01.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-13T14:54:01.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-13T14:54:01.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-13T14:54:01.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-09-13T14:54:02.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-13T14:54:02.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-13T14:54:02.948 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'ecpool' created 2024-09-13T14:54:02.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:54:03.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-13T14:54:03.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-13T14:54:04.630 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 2 allow_ec_overwrites to true 2024-09-13T14:54:04.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-13T14:54:04.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:54:04.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:54:04.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:54:04.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:54:04.657 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:54:04.658 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:54:04.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:54:04.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:54:04.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:54:04.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:54:04.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:54:04.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:54:04.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:54:04.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:54:04.779 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:54:05.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:54:05.131 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:54:05.131 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:54:05.132 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:54:05.132 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:54:05.132 INFO:tasks.workunit.client.0.smithi112.stderr:5 2024-09-13T14:54:05.132 INFO:tasks.workunit.client.0.smithi112.stderr:6 2024-09-13T14:54:05.132 INFO:tasks.workunit.client.0.smithi112.stderr:7 2024-09-13T14:54:05.132 INFO:tasks.workunit.client.0.smithi112.stderr:8 2024-09-13T14:54:05.132 INFO:tasks.workunit.client.0.smithi112.stderr:9' 2024-09-13T14:54:05.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:54:05.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:05.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:54:05.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836553 2024-09-13T14:54:05.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836553 2024-09-13T14:54:05.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553' 2024-09-13T14:54:05.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:05.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:54:05.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673026 2024-09-13T14:54:05.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673026 2024-09-13T14:54:05.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673026' 2024-09-13T14:54:05.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:05.494 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:54:05.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509498 2024-09-13T14:54:05.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509498 2024-09-13T14:54:05.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673026 2-64424509498' 2024-09-13T14:54:05.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:05.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:54:05.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345973 2024-09-13T14:54:05.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345973 2024-09-13T14:54:05.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673026 2-64424509498 3-85899345973' 2024-09-13T14:54:05.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:05.889 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:54:06.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182446 2024-09-13T14:54:06.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182446 2024-09-13T14:54:06.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673026 2-64424509498 3-85899345973 4-107374182446' 2024-09-13T14:54:06.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:06.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:54:06.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018917 2024-09-13T14:54:06.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018917 2024-09-13T14:54:06.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673026 2-64424509498 3-85899345973 4-107374182446 5-128849018917' 2024-09-13T14:54:06.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:06.167 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:54:06.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855391 2024-09-13T14:54:06.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855391 2024-09-13T14:54:06.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673026 2-64424509498 3-85899345973 4-107374182446 5-128849018917 6-150323855391' 2024-09-13T14:54:06.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:06.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:54:06.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691863 2024-09-13T14:54:06.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691863 2024-09-13T14:54:06.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673026 2-64424509498 3-85899345973 4-107374182446 5-128849018917 6-150323855391 7-171798691863' 2024-09-13T14:54:06.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:06.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:54:06.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528340 2024-09-13T14:54:06.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528340 2024-09-13T14:54:06.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673026 2-64424509498 3-85899345973 4-107374182446 5-128849018917 6-150323855391 7-171798691863 8-193273528340' 2024-09-13T14:54:06.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:06.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:54:06.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364812 2024-09-13T14:54:06.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364812 2024-09-13T14:54:06.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-42949673026 2-64424509498 3-85899345973 4-107374182446 5-128849018917 6-150323855391 7-171798691863 8-193273528340 9-214748364812' 2024-09-13T14:54:06.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:06.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836553 2024-09-13T14:54:06.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:06.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:54:06.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836553 2024-09-13T14:54:06.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:06.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836553 2024-09-13T14:54:06.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-09-13T14:54:06.797 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836553 2024-09-13T14:54:06.797 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:54:07.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836554 -lt 21474836553 2024-09-13T14:54:07.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:07.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673026 2024-09-13T14:54:07.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:07.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:54:07.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673026 2024-09-13T14:54:07.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:07.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673026 2024-09-13T14:54:07.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673026' 2024-09-13T14:54:07.113 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673026 2024-09-13T14:54:07.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:54:07.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673026 -lt 42949673026 2024-09-13T14:54:07.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:07.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509498 2024-09-13T14:54:07.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:07.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:54:07.419 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509498 2024-09-13T14:54:07.420 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:07.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509498 2024-09-13T14:54:07.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509498' 2024-09-13T14:54:07.421 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509498 2024-09-13T14:54:07.422 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:54:07.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509499 -lt 64424509498 2024-09-13T14:54:07.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:07.746 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345973 2024-09-13T14:54:07.746 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:07.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:54:07.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345973 2024-09-13T14:54:07.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:07.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345973 2024-09-13T14:54:07.750 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.3 seq 85899345973 2024-09-13T14:54:07.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345973' 2024-09-13T14:54:07.751 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:54:08.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345974 -lt 85899345973 2024-09-13T14:54:08.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:08.252 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:08.252 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182446 2024-09-13T14:54:08.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:54:08.254 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182446 2024-09-13T14:54:08.254 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:08.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182446 2024-09-13T14:54:08.256 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.4 seq 107374182446 2024-09-13T14:54:08.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182446' 2024-09-13T14:54:08.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:54:08.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182447 -lt 107374182446 2024-09-13T14:54:08.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:08.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018917 2024-09-13T14:54:08.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:08.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:54:08.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018917 2024-09-13T14:54:08.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:08.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018917 2024-09-13T14:54:08.637 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.5 seq 128849018917 2024-09-13T14:54:08.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018917' 2024-09-13T14:54:08.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:54:08.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018918 -lt 128849018917 2024-09-13T14:54:08.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:08.969 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855391 2024-09-13T14:54:08.969 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:08.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:54:08.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855391 2024-09-13T14:54:08.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:08.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855391 2024-09-13T14:54:08.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855391' 2024-09-13T14:54:08.973 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.6 seq 150323855391 2024-09-13T14:54:08.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:54:09.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855392 -lt 150323855391 2024-09-13T14:54:09.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:09.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691863 2024-09-13T14:54:09.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:09.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:54:09.320 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691863 2024-09-13T14:54:09.320 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:09.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691863 2024-09-13T14:54:09.321 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.7 seq 171798691863 2024-09-13T14:54:09.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691863' 2024-09-13T14:54:09.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:54:09.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691864 -lt 171798691863 2024-09-13T14:54:09.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:09.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528340 2024-09-13T14:54:09.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:09.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:54:09.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528340 2024-09-13T14:54:09.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:09.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528340 2024-09-13T14:54:09.664 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.8 seq 193273528340 2024-09-13T14:54:09.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528340' 2024-09-13T14:54:09.665 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:54:10.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528340 -lt 193273528340 2024-09-13T14:54:10.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:10.005 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364812 2024-09-13T14:54:10.005 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:10.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:54:10.008 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364812 2024-09-13T14:54:10.008 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:10.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364812 2024-09-13T14:54:10.009 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.9 seq 214748364812 2024-09-13T14:54:10.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364812' 2024-09-13T14:54:10.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:54:10.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364813 -lt 214748364812 2024-09-13T14:54:10.339 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:54:10.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:54:10.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:54:10.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:54:10.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:54:10.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:54:10.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:54:10.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:54:10.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:54:10.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:54:10.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:54:11.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:54:11.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:54:11.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:54:11.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:54:11.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:54:11.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:54:11.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:54:11.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-13T14:54:11.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-13T14:54:11.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-13T14:54:11.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-13T14:54:11.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-13T14:54:11.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T14:54:11.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T14:54:11.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-13T14:54:11.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T14:54:11.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T14:54:11.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T14:54:11.880 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T14:54:11.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T14:54:12.542 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T14:54:12.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T14:54:12.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T14:54:12.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-13T14:54:13.043 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-13T14:54:13.043 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-13T14:54:13.043 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:54:13.043 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:54:13.043 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:54:13.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-13T14:54:13.467 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-13T14:54:13.467 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-13T14:54:13.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-13T14:54:13.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-13T14:54:13.468 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:54:13.468 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T14:54:13.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-13T14:54:13.814 INFO:tasks.workunit.client.0.smithi112.stderr:5 2024-09-13T14:54:13.814 INFO:tasks.workunit.client.0.smithi112.stderr:9 2024-09-13T14:54:13.815 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:54:13.815 INFO:tasks.workunit.client.0.smithi112.stderr:6 2024-09-13T14:54:13.815 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:54:13.815 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:54:13.815 INFO:tasks.workunit.client.0.smithi112.stderr:7' 2024-09-13T14:54:13.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 6 2 1 7 2024-09-13T14:54:13.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-09-13T14:54:13.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-13T14:54:13.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-09-13T14:54:13.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-09-13T14:54:13.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-13T14:54:13.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-13T14:54:13.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-13T14:54:13.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-13T14:54:13.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:54:13.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:54:13.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:54:13.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:54:13.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:54:13.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:54:13.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:54:13.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:54:13.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:54:13.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:54:13.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:54:13.819 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:54:13.819 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:54:13.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:54:13.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:54:13.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:54:14.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:54:14.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:54:14.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:54:14.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:54:14.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:54:14.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:54:14.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:54:14.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-13T14:54:15.941 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-13T14:54:16.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:54:16.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:54:16.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:54:16.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:54:16.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:54:16.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:54:16.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:54:16.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:54:16.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:54:16.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:54:16.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:54:16.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:54:16.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:54:16.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:54:16.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:54:16.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:54:16.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:54:16.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:54:16.481 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:54:16.481 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:54:16.481 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:54:16.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:54:16.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:54:16.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:54:16.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:54:16.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:54:16.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:54:16.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:54:16.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:54:16.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:54:16.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:54:16.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:54:16.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:54:16.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:54:16.483 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.3 2024-09-13T14:54:16.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:54:16.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:54:16.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:54:16.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:54:16.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:54:16.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:54:16.518 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:54:16.517+0000 7fd082aa17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:54:16.521 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:54:16.517+0000 7fd082aa17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:54:16.522 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:54:16.521+0000 7fd082aa17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:54:16.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:54:16.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:54:16.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:54:16.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:54:16.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:54:16.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:54:16.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:54:16.862 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:54:16.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:54:16.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:54:17.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:54:18.037 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:54:18.033+0000 7fd082aa17c0 -1 Falling back to public interface 2024-09-13T14:54:18.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:54:18.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:54:18.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:54:18.205 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:54:18.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:54:18.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:54:18.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:54:19.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:54:19.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:54:19.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:54:19.537 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:54:19.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:54:19.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:54:19.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:54:20.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:54:20.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:54:20.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:54:20.873 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:54:20.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:54:20.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:54:21.091 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:54:21.089+0000 7fd082aa17c0 -1 osd.3 68 log_to_monitors true 2024-09-13T14:54:21.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:54:22.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:54:22.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:54:22.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:54:22.295 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:54:22.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:54:22.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:54:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:osd.3 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [20,68) [v2:127.0.0.1:6826/2031008543,v1:127.0.0.1:6827/2031008543] [v2:127.0.0.1:6828/2031008543,v1:127.0.0.1:6829/2031008543] exists,up 858e19ad-bbe8-4eeb-8da5-f6f455b0cb32 2024-09-13T14:54:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:54:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:54:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:54:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:54:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:54:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:54:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:54:22.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:54:22.726 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:54:22.727 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:54:22.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:54:22.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:54:22.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:54:22.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:54:22.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:54:22.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:54:22.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:54:22.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:54:22.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:54:23.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:54:23.205 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:54:23.205 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:54:23.205 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:54:23.205 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:54:23.205 INFO:tasks.workunit.client.0.smithi112.stderr:5 2024-09-13T14:54:23.205 INFO:tasks.workunit.client.0.smithi112.stderr:6 2024-09-13T14:54:23.205 INFO:tasks.workunit.client.0.smithi112.stderr:7 2024-09-13T14:54:23.205 INFO:tasks.workunit.client.0.smithi112.stderr:8 2024-09-13T14:54:23.206 INFO:tasks.workunit.client.0.smithi112.stderr:9' 2024-09-13T14:54:23.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:54:23.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:23.206 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:54:23.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836558 2024-09-13T14:54:23.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836558 2024-09-13T14:54:23.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558' 2024-09-13T14:54:23.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:23.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:54:23.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673031 2024-09-13T14:54:23.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673031 2024-09-13T14:54:23.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673031' 2024-09-13T14:54:23.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:23.518 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:54:23.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509503 2024-09-13T14:54:23.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509503 2024-09-13T14:54:23.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509503' 2024-09-13T14:54:23.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:23.650 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:54:23.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645314 2024-09-13T14:54:23.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645314 2024-09-13T14:54:23.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509503 3-309237645314' 2024-09-13T14:54:23.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:23.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:54:23.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182451 2024-09-13T14:54:23.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182451 2024-09-13T14:54:23.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509503 3-309237645314 4-107374182451' 2024-09-13T14:54:23.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:23.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:54:24.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018922 2024-09-13T14:54:24.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018922 2024-09-13T14:54:24.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509503 3-309237645314 4-107374182451 5-128849018922' 2024-09-13T14:54:24.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:24.084 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:54:24.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855396 2024-09-13T14:54:24.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855396 2024-09-13T14:54:24.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509503 3-309237645314 4-107374182451 5-128849018922 6-150323855396' 2024-09-13T14:54:24.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:24.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:54:24.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691868 2024-09-13T14:54:24.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691868 2024-09-13T14:54:24.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509503 3-309237645314 4-107374182451 5-128849018922 6-150323855396 7-171798691868' 2024-09-13T14:54:24.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:24.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:54:24.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528345 2024-09-13T14:54:24.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528345 2024-09-13T14:54:24.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509503 3-309237645314 4-107374182451 5-128849018922 6-150323855396 7-171798691868 8-193273528345' 2024-09-13T14:54:24.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:24.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:54:24.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364817 2024-09-13T14:54:24.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364817 2024-09-13T14:54:24.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673031 2-64424509503 3-309237645314 4-107374182451 5-128849018922 6-150323855396 7-171798691868 8-193273528345 9-214748364817' 2024-09-13T14:54:24.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:24.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836558 2024-09-13T14:54:24.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:24.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:54:24.704 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836558 2024-09-13T14:54:24.704 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:24.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836558 2024-09-13T14:54:24.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2024-09-13T14:54:24.704 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836558 2024-09-13T14:54:24.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:54:25.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836558 -lt 21474836558 2024-09-13T14:54:25.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:25.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673031 2024-09-13T14:54:25.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:25.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:54:25.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673031 2024-09-13T14:54:25.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:25.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673031 2024-09-13T14:54:25.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2024-09-13T14:54:25.026 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949673031 2024-09-13T14:54:25.027 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:54:25.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673031 -lt 42949673031 2024-09-13T14:54:25.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:25.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:25.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509503 2024-09-13T14:54:25.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:54:25.359 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509503 2024-09-13T14:54:25.359 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:25.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509503 2024-09-13T14:54:25.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2024-09-13T14:54:25.361 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509503 2024-09-13T14:54:25.361 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:54:25.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509503 -lt 64424509503 2024-09-13T14:54:25.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:25.689 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645314 2024-09-13T14:54:25.690 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:25.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:54:25.692 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645314 2024-09-13T14:54:25.692 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:25.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645314 2024-09-13T14:54:25.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645314' 2024-09-13T14:54:25.693 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 309237645314 2024-09-13T14:54:25.693 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:54:26.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645314 -lt 309237645314 2024-09-13T14:54:26.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:26.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:26.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182451 2024-09-13T14:54:26.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:54:26.043 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182451 2024-09-13T14:54:26.043 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:26.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182451 2024-09-13T14:54:26.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2024-09-13T14:54:26.045 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.4 seq 107374182451 2024-09-13T14:54:26.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:54:26.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182451 -lt 107374182451 2024-09-13T14:54:26.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:26.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018922 2024-09-13T14:54:26.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:26.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:54:26.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018922 2024-09-13T14:54:26.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:26.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018922 2024-09-13T14:54:26.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018922' 2024-09-13T14:54:26.416 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.5 seq 128849018922 2024-09-13T14:54:26.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:54:26.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018922 -lt 128849018922 2024-09-13T14:54:26.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:26.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:26.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855396 2024-09-13T14:54:26.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:54:26.746 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855396 2024-09-13T14:54:26.746 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:26.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855396 2024-09-13T14:54:26.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855396' 2024-09-13T14:54:26.748 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.6 seq 150323855396 2024-09-13T14:54:26.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:54:27.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855396 -lt 150323855396 2024-09-13T14:54:27.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:27.053 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691868 2024-09-13T14:54:27.053 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:27.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:54:27.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691868 2024-09-13T14:54:27.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:27.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691868 2024-09-13T14:54:27.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691868' 2024-09-13T14:54:27.057 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.7 seq 171798691868 2024-09-13T14:54:27.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:54:27.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691868 -lt 171798691868 2024-09-13T14:54:27.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:27.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528345 2024-09-13T14:54:27.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:27.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:54:27.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528345 2024-09-13T14:54:27.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:27.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528345 2024-09-13T14:54:27.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528345' 2024-09-13T14:54:27.396 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.8 seq 193273528345 2024-09-13T14:54:27.396 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:54:27.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528345 -lt 193273528345 2024-09-13T14:54:27.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:27.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364817 2024-09-13T14:54:27.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:27.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:54:27.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364817 2024-09-13T14:54:27.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:27.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364817 2024-09-13T14:54:27.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364817' 2024-09-13T14:54:27.725 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.9 seq 214748364817 2024-09-13T14:54:27.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:54:28.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364817 -lt 214748364817 2024-09-13T14:54:28.065 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:54:28.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:54:28.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:54:28.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:54:28.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:54:28.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:54:28.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:54:28.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:54:28.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:54:28.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:54:28.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:54:28.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:54:28.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:54:28.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:54:28.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:54:29.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:54:29.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:54:29.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:54:29.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-13T14:54:29.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:54:29.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:54:29.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:54:29.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:54:29.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-13T14:54:29.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-13T14:54:29.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:54:29.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:54:29.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:54:29.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:54:29.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:54:29.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:54:29.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:54:02.806028+0000 2024-09-13T14:54:29.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:54:30.151 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:54:30.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:54:02.806028+0000 2024-09-13T14:54:30.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:54:30.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:54:02.806028+0000 2024-09-13T14:54:30.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:54:30.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:54:30.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:54:30.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:54:30.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:54:30.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:54:30.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:54:30.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:54:30.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:54:02.806028+0000 '>' 2024-09-13T14:54:02.806028+0000 2024-09-13T14:54:30.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:54:31.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:54:31.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:54:31.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:54:31.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:54:31.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:54:31.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:54:31.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:54:31.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:54:02.806028+0000 '>' 2024-09-13T14:54:02.806028+0000 2024-09-13T14:54:31.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:54:32.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:54:32.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:54:32.822 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:54:32.822 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:54:32.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:54:32.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:54:32.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:54:33.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:54:30.219217+0000 '>' 2024-09-13T14:54:02.806028+0000 2024-09-13T14:54:33.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:54:33.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:54:35.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:54:35.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:54:35.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:54:35.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-13T14:54:35.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:54:35.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:54:35.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:54:35.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:54:35.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-13T14:54:35.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:54:35.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-13T14:54:35.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:54:35.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:54:35.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:54:35.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:54:35.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:54:35.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:54:35.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:54:35.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:54:35.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:54:35.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-13T14:54:35.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:54:35.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:54:35.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-13T14:54:36.037 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:54:36.037 INFO:tasks.workunit.client.0.smithi112.stdout:hinfo_key 2024-09-13T14:54:36.037 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:54:36.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-13T14:54:36.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:54:36.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:54:36.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-13T14:54:36.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:54:36.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:54:36.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:54:36.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:54:36.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:54:36.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:54:36.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-13T14:54:36.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-13T14:54:36.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:54:36.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:54:36.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:54:36.364 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:54:36.364 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:54:36.364 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:54:36.364 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:54:36.364 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:54:36.364 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:54:36.364 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:54:36.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:54:36.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:54:36.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:54:36.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:54:36.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:54:36.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:54:36.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:54:36.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:54:36.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:54:36.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:54:36.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-13T14:54:36.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-13T14:54:36.367 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.9 2024-09-13T14:54:36.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:54:36.367 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-13T14:54:36.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-13T14:54:36.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:54:36.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:54:36.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:54:36.383 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:54:36.380+0000 7f57c712a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:54:36.474 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:54:36.468+0000 7f57c712a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:54:36.475 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:54:36.472+0000 7f57c712a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:54:36.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-13T14:54:36.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:54:36.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-13T14:54:36.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:54:36.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:54:36.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:54:36.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:54:36.726 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:54:36.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:54:36.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:54:37.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:54:37.797 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:54:37.792+0000 7f57c712a7c0 -1 Falling back to public interface 2024-09-13T14:54:38.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:54:38.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:54:38.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:54:38.076 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:54:38.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:54:38.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:54:38.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:54:39.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:54:39.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:54:39.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:54:39.408 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:54:39.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:54:39.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:54:39.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:54:40.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:54:40.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:54:40.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:54:40.726 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:54:40.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:54:40.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:54:41.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:54:41.151 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:54:41.148+0000 7f57c712a7c0 -1 osd.9 73 log_to_monitors true 2024-09-13T14:54:42.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:54:42.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:54:42.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:54:42.061 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:54:42.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:54:42.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:54:42.542 INFO:tasks.workunit.client.0.smithi112.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 74 last_clean_interval [50,73) [v2:127.0.0.1:6874/3731045535,v1:127.0.0.1:6875/3731045535] [v2:127.0.0.1:6876/3731045535,v1:127.0.0.1:6877/3731045535] exists,up 0253c251-b81a-4f43-af92-c81fb0549c27 2024-09-13T14:54:42.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:54:42.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:54:42.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:54:42.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:54:42.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:54:42.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:54:42.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:54:42.544 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:54:42.544 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:54:42.544 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:54:42.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:54:42.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:54:42.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:54:42.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:54:42.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:54:42.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:54:42.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:54:42.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:54:42.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:54:43.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:54:43.103 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:54:43.103 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:54:43.104 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:54:43.104 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:54:43.104 INFO:tasks.workunit.client.0.smithi112.stderr:5 2024-09-13T14:54:43.104 INFO:tasks.workunit.client.0.smithi112.stderr:6 2024-09-13T14:54:43.104 INFO:tasks.workunit.client.0.smithi112.stderr:7 2024-09-13T14:54:43.104 INFO:tasks.workunit.client.0.smithi112.stderr:8 2024-09-13T14:54:43.104 INFO:tasks.workunit.client.0.smithi112.stderr:9' 2024-09-13T14:54:43.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:54:43.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:43.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:54:43.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836563 2024-09-13T14:54:43.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836563 2024-09-13T14:54:43.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563' 2024-09-13T14:54:43.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:43.281 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:54:43.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673036 2024-09-13T14:54:43.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673036 2024-09-13T14:54:43.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673036' 2024-09-13T14:54:43.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:43.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:54:43.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509508 2024-09-13T14:54:43.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509508 2024-09-13T14:54:43.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509508' 2024-09-13T14:54:43.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:43.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:54:43.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645319 2024-09-13T14:54:43.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645319 2024-09-13T14:54:43.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509508 3-309237645319' 2024-09-13T14:54:43.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:43.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:54:43.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182456 2024-09-13T14:54:43.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182456 2024-09-13T14:54:43.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509508 3-309237645319 4-107374182456' 2024-09-13T14:54:43.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:43.907 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:54:44.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018927 2024-09-13T14:54:44.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018927 2024-09-13T14:54:44.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509508 3-309237645319 4-107374182456 5-128849018927' 2024-09-13T14:54:44.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:44.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:54:44.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855401 2024-09-13T14:54:44.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855401 2024-09-13T14:54:44.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509508 3-309237645319 4-107374182456 5-128849018927 6-150323855401' 2024-09-13T14:54:44.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:44.194 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:54:44.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691873 2024-09-13T14:54:44.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691873 2024-09-13T14:54:44.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509508 3-309237645319 4-107374182456 5-128849018927 6-150323855401 7-171798691873' 2024-09-13T14:54:44.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:44.382 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:54:44.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528350 2024-09-13T14:54:44.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528350 2024-09-13T14:54:44.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509508 3-309237645319 4-107374182456 5-128849018927 6-150323855401 7-171798691873 8-193273528350' 2024-09-13T14:54:44.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:44.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:54:44.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=330712481794 2024-09-13T14:54:44.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 330712481794 2024-09-13T14:54:44.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673036 2-64424509508 3-309237645319 4-107374182456 5-128849018927 6-150323855401 7-171798691873 8-193273528350 9-330712481794' 2024-09-13T14:54:44.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:44.683 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836563 2024-09-13T14:54:44.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:44.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:54:44.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836563 2024-09-13T14:54:44.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:44.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836563 2024-09-13T14:54:44.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836563' 2024-09-13T14:54:44.686 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836563 2024-09-13T14:54:44.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:54:45.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836563 -lt 21474836563 2024-09-13T14:54:45.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:45.011 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673036 2024-09-13T14:54:45.011 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:45.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:54:45.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673036 2024-09-13T14:54:45.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:45.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673036 2024-09-13T14:54:45.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673036' 2024-09-13T14:54:45.015 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949673036 2024-09-13T14:54:45.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:54:45.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673036 -lt 42949673036 2024-09-13T14:54:45.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:45.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509508 2024-09-13T14:54:45.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:45.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:54:45.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509508 2024-09-13T14:54:45.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:45.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509508 2024-09-13T14:54:45.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509508' 2024-09-13T14:54:45.362 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509508 2024-09-13T14:54:45.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:54:45.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509508 -lt 64424509508 2024-09-13T14:54:45.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:45.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645319 2024-09-13T14:54:45.716 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:45.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:54:45.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645319 2024-09-13T14:54:45.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:45.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645319 2024-09-13T14:54:45.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645319' 2024-09-13T14:54:45.719 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 309237645319 2024-09-13T14:54:45.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:54:46.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645319 -lt 309237645319 2024-09-13T14:54:46.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:46.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182456 2024-09-13T14:54:46.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:46.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:54:46.075 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182456 2024-09-13T14:54:46.075 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:46.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182456 2024-09-13T14:54:46.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182456' 2024-09-13T14:54:46.077 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.4 seq 107374182456 2024-09-13T14:54:46.077 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:54:46.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182456 -lt 107374182456 2024-09-13T14:54:46.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:46.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018927 2024-09-13T14:54:46.442 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:46.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:54:46.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018927 2024-09-13T14:54:46.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:46.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018927 2024-09-13T14:54:46.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018927' 2024-09-13T14:54:46.445 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.5 seq 128849018927 2024-09-13T14:54:46.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:54:46.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018927 -lt 128849018927 2024-09-13T14:54:46.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:46.771 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855401 2024-09-13T14:54:46.771 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:46.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:54:46.773 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855401 2024-09-13T14:54:46.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855401 2024-09-13T14:54:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855401' 2024-09-13T14:54:46.775 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.6 seq 150323855401 2024-09-13T14:54:46.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:54:47.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855401 -lt 150323855401 2024-09-13T14:54:47.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:47.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691873 2024-09-13T14:54:47.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:47.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:54:47.082 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691873 2024-09-13T14:54:47.082 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:47.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691873 2024-09-13T14:54:47.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691873' 2024-09-13T14:54:47.083 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.7 seq 171798691873 2024-09-13T14:54:47.083 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:54:47.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691873 -lt 171798691873 2024-09-13T14:54:47.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:47.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528350 2024-09-13T14:54:47.429 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:47.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:54:47.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528350 2024-09-13T14:54:47.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:47.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528350 2024-09-13T14:54:47.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528350' 2024-09-13T14:54:47.432 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.8 seq 193273528350 2024-09-13T14:54:47.433 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:54:47.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528350 -lt 193273528350 2024-09-13T14:54:47.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:54:47.754 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-330712481794 2024-09-13T14:54:47.754 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:54:47.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:54:47.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-330712481794 2024-09-13T14:54:47.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:54:47.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=330712481794 2024-09-13T14:54:47.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-09-13T14:54:47.757 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.9 seq 330712481794 2024-09-13T14:54:47.758 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:54:48.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 330712481794 -lt 330712481794 2024-09-13T14:54:48.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:54:48.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:54:48.088 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:54:48.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:54:48.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:54:48.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:54:48.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:54:48.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:54:48.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:54:48.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:54:48.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:54:48.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:54:48.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:54:48.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:54:48.845 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:54:49.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:54:49.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:54:49.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:54:49.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:54:49.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:54:49.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-09-13T14:54:49.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-13T14:54:49.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-13T14:54:49.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-09-13T14:54:49.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:54:49.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:54:49.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:54:49.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-13T14:54:49.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:54:49.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:54:49.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:54:49.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:54:49.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-13T14:54:49.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:54:49.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-13T14:54:49.285 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:54:49.285 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:54:49.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:54:49.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:54:49.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:54:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:54:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:54:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:54:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:54:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-13T14:54:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:54:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:54:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-13T14:54:51.606 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-13T14:54:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-13T14:54:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:54:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:54:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-13T14:54:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:54:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:54:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:54:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:54:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:54:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:54:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-13T14:54:52.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-13T14:54:52.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:54:52.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:54:52.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:54:52.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:54:52.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:54:52.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:54:52.148 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:54:52.148 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:54:52.148 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:54:52.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:54:52.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:54:52.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:54:52.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:54:52.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:54:52.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:54:52.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:54:52.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:54:52.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:54:52.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:54:52.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:54:52.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-13T14:54:52.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-13T14:54:52.150 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.5 2024-09-13T14:54:52.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:54:52.150 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-13T14:54:52.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-13T14:54:52.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:54:52.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:54:52.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:54:52.190 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:54:52.188+0000 7f14bce2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:54:52.205 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:54:52.204+0000 7f14bce2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:54:52.206 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:54:52.208+0000 7f14bce2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:54:52.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-13T14:54:52.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:54:52.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-13T14:54:52.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:54:52.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:54:52.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:54:52.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:54:52.493 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:54:52.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:54:52.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:54:52.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:54:53.717 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:54:53.716+0000 7f14bce2d7c0 -1 Falling back to public interface 2024-09-13T14:54:53.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:54:53.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:54:53.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:54:53.832 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:54:53.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:54:53.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:54:54.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:54:55.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:54:55.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:54:55.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:54:55.172 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:54:55.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:54:55.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:54:55.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:54:56.482 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:54:56.480+0000 7f14bce2d7c0 -1 osd.5 79 log_to_monitors true 2024-09-13T14:54:56.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:54:56.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:54:56.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:54:56.511 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:54:56.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:54:56.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:54:56.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:54:57.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:54:57.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:54:57.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:54:57.885 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:54:57.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:54:57.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:54:58.477 INFO:tasks.workunit.client.0.smithi112.stderr:osd.5 up in weight 1 up_from 83 up_thru 83 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/2205711793,v1:127.0.0.1:6843/2205711793] [v2:127.0.0.1:6844/2205711793,v1:127.0.0.1:6845/2205711793] exists,up 3df67c99-00a1-4d12-b468-6f9b9faac1b1 2024-09-13T14:54:58.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:54:58.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:54:58.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:54:58.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:54:58.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:54:58.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:54:58.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:54:58.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:54:58.479 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:54:58.479 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:54:58.479 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:54:58.479 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:54:58.479 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:54:58.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:54:58.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:54:58.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:54:58.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:54:58.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:54:58.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:54:58.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:54:58.949 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:54:58.949 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:54:58.949 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:54:58.950 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:54:58.950 INFO:tasks.workunit.client.0.smithi112.stderr:5 2024-09-13T14:54:58.950 INFO:tasks.workunit.client.0.smithi112.stderr:6 2024-09-13T14:54:58.950 INFO:tasks.workunit.client.0.smithi112.stderr:7 2024-09-13T14:54:58.950 INFO:tasks.workunit.client.0.smithi112.stderr:8 2024-09-13T14:54:58.950 INFO:tasks.workunit.client.0.smithi112.stderr:9' 2024-09-13T14:54:58.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:54:58.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:58.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:54:59.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836567 2024-09-13T14:54:59.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836567 2024-09-13T14:54:59.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567' 2024-09-13T14:54:59.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:59.098 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:54:59.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673040 2024-09-13T14:54:59.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673040 2024-09-13T14:54:59.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040' 2024-09-13T14:54:59.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:59.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:54:59.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509512 2024-09-13T14:54:59.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509512 2024-09-13T14:54:59.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509512' 2024-09-13T14:54:59.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:59.382 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:54:59.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645323 2024-09-13T14:54:59.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645323 2024-09-13T14:54:59.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509512 3-309237645323' 2024-09-13T14:54:59.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:59.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:54:59.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182460 2024-09-13T14:54:59.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182460 2024-09-13T14:54:59.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509512 3-309237645323 4-107374182460' 2024-09-13T14:54:59.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:59.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:54:59.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=356482285570 2024-09-13T14:54:59.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 356482285570 2024-09-13T14:54:59.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509512 3-309237645323 4-107374182460 5-356482285570' 2024-09-13T14:54:59.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:59.791 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:54:59.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855405 2024-09-13T14:54:59.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855405 2024-09-13T14:54:59.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509512 3-309237645323 4-107374182460 5-356482285570 6-150323855405' 2024-09-13T14:54:59.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:54:59.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:55:00.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691877 2024-09-13T14:55:00.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691877 2024-09-13T14:55:00.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509512 3-309237645323 4-107374182460 5-356482285570 6-150323855405 7-171798691877' 2024-09-13T14:55:00.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:00.064 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:55:00.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528354 2024-09-13T14:55:00.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528354 2024-09-13T14:55:00.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509512 3-309237645323 4-107374182460 5-356482285570 6-150323855405 7-171798691877 8-193273528354' 2024-09-13T14:55:00.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:00.198 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:55:00.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=330712481798 2024-09-13T14:55:00.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 330712481798 2024-09-13T14:55:00.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673040 2-64424509512 3-309237645323 4-107374182460 5-356482285570 6-150323855405 7-171798691877 8-193273528354 9-330712481798' 2024-09-13T14:55:00.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:00.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836567 2024-09-13T14:55:00.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:00.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:55:00.339 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836567 2024-09-13T14:55:00.339 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:00.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836567 2024-09-13T14:55:00.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2024-09-13T14:55:00.340 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836567 2024-09-13T14:55:00.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:55:00.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836567 -lt 21474836567 2024-09-13T14:55:00.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:00.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673040 2024-09-13T14:55:00.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:00.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:55:00.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673040 2024-09-13T14:55:00.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:00.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673040 2024-09-13T14:55:00.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-09-13T14:55:00.650 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949673040 2024-09-13T14:55:00.650 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:55:01.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673040 -lt 42949673040 2024-09-13T14:55:01.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:01.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509512 2024-09-13T14:55:01.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:01.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:55:01.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509512 2024-09-13T14:55:01.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:01.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509512 2024-09-13T14:55:01.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2024-09-13T14:55:01.029 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509512 2024-09-13T14:55:01.029 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:55:01.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509512 -lt 64424509512 2024-09-13T14:55:01.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:01.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645323 2024-09-13T14:55:01.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:01.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:55:01.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645323 2024-09-13T14:55:01.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:01.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645323 2024-09-13T14:55:01.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645323' 2024-09-13T14:55:01.405 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 309237645323 2024-09-13T14:55:01.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:55:01.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645323 -lt 309237645323 2024-09-13T14:55:01.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:01.728 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:01.728 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182460 2024-09-13T14:55:01.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:55:01.731 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182460 2024-09-13T14:55:01.731 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:01.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182460 2024-09-13T14:55:01.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182460' 2024-09-13T14:55:01.733 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.4 seq 107374182460 2024-09-13T14:55:01.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:55:02.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182460 -lt 107374182460 2024-09-13T14:55:02.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:02.107 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:02.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-356482285570 2024-09-13T14:55:02.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:55:02.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-356482285570 2024-09-13T14:55:02.110 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:02.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=356482285570 2024-09-13T14:55:02.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 356482285570' 2024-09-13T14:55:02.111 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.5 seq 356482285570 2024-09-13T14:55:02.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:55:02.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 356482285571 -lt 356482285570 2024-09-13T14:55:02.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:02.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:02.518 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855405 2024-09-13T14:55:02.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:55:02.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855405 2024-09-13T14:55:02.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:02.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855405 2024-09-13T14:55:02.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855405' 2024-09-13T14:55:02.521 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.6 seq 150323855405 2024-09-13T14:55:02.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:55:02.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855405 -lt 150323855405 2024-09-13T14:55:02.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:02.821 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691877 2024-09-13T14:55:02.821 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:02.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:55:02.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691877 2024-09-13T14:55:02.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:02.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691877 2024-09-13T14:55:02.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691877' 2024-09-13T14:55:02.827 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.7 seq 171798691877 2024-09-13T14:55:02.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:55:03.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691877 -lt 171798691877 2024-09-13T14:55:03.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:03.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528354 2024-09-13T14:55:03.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:03.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:55:03.128 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528354 2024-09-13T14:55:03.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:03.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528354 2024-09-13T14:55:03.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528354' 2024-09-13T14:55:03.130 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.8 seq 193273528354 2024-09-13T14:55:03.131 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:55:03.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528354 -lt 193273528354 2024-09-13T14:55:03.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:03.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-330712481798 2024-09-13T14:55:03.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:03.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:55:03.455 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-330712481798 2024-09-13T14:55:03.455 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:03.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=330712481798 2024-09-13T14:55:03.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-09-13T14:55:03.457 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.9 seq 330712481798 2024-09-13T14:55:03.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:55:03.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 330712481799 -lt 330712481798 2024-09-13T14:55:03.846 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:55:03.847 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:55:03.847 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:55:04.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:55:04.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:55:04.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:55:04.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:55:04.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:55:04.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:55:04.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:55:04.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:55:04.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:55:04.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:55:04.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:55:04.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:55:04.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:55:04.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:55:04.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:55:04.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-13T14:55:04.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:55:04.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:55:04.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:55:04.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:55:05.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-13T14:55:05.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-13T14:55:05.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:55:05.312 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:55:05.312 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:55:05.312 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:55:05.312 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:55:05.312 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:55:05.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:54:30.219217+0000 2024-09-13T14:55:05.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:55:05.911 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:55:05.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:54:30.219217+0000 2024-09-13T14:55:05.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:55:05.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:54:30.219217+0000 2024-09-13T14:55:05.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:55:05.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:55:05.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:55:05.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:55:05.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:55:05.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:55:05.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:55:05.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:55:06.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:54:30.219217+0000 '>' 2024-09-13T14:54:30.219217+0000 2024-09-13T14:55:06.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:55:07.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:55:07.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:55:07.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:55:07.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:55:07.263 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:55:07.263 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:55:07.263 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:55:07.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:54:30.219217+0000 '>' 2024-09-13T14:54:30.219217+0000 2024-09-13T14:55:07.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:55:08.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:55:08.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:55:08.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:55:08.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:55:08.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:55:08.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:55:08.603 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:55:08.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:55:06.106012+0000 '>' 2024-09-13T14:54:30.219217+0000 2024-09-13T14:55:08.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:55:08.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:55:10.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:55:10.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:55:10.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:55:10.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-13T14:55:10.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:55:10.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:55:10.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:55:10.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:55:10.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-13T14:55:10.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:55:10.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-13T14:55:10.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:55:10.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:55:10.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:55:10.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:55:10.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:55:11.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:55:11.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:55:11.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:55:11.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:55:11.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-13T14:55:11.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:55:11.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:55:11.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-13T14:55:11.788 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:55:11.788 INFO:tasks.workunit.client.0.smithi112.stdout:hinfo_key 2024-09-13T14:55:11.788 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:55:12.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-13T14:55:12.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:55:12.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:55:12.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-13T14:55:12.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:55:12.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:55:12.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:55:12.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:55:12.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:55:12.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:55:12.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-13T14:55:12.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-13T14:55:12.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:55:12.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:55:12.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:55:12.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:55:12.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:55:12.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:55:12.087 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:55:12.087 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:55:12.087 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:55:12.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:55:12.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:55:12.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:55:12.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:55:12.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:55:12.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:55:12.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:55:12.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:55:12.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:55:12.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:55:12.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:55:12.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-13T14:55:12.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-13T14:55:12.088 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.9 2024-09-13T14:55:12.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:55:12.089 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-13T14:55:12.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-13T14:55:12.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:55:12.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:55:12.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:55:12.135 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:55:12.135+0000 7f5cc8d677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:55:12.166 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:55:12.163+0000 7f5cc8d677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:55:12.168 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:55:12.167+0000 7f5cc8d677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:55:12.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-13T14:55:12.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:55:12.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-13T14:55:12.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:55:12.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:55:12.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:12.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:55:12.455 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:55:12.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:12.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:55:13.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:55:13.269 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:55:13.267+0000 7f5cc8d677c0 -1 Falling back to public interface 2024-09-13T14:55:14.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:55:14.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:14.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:55:14.039 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:55:14.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:14.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:55:14.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:55:15.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:55:15.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:15.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:55:15.375 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:55:15.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:15.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:55:15.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:55:16.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:55:16.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:16.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:55:16.708 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:55:16.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:16.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:55:16.951 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:55:16.951+0000 7f5cc8d677c0 -1 osd.9 84 log_to_monitors true 2024-09-13T14:55:17.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:55:18.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:55:18.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:18.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:55:18.093 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:55:18.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:18.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:55:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:osd.9 up in weight 1 up_from 89 up_thru 0 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6874/3178977884,v1:127.0.0.1:6875/3178977884] [v2:127.0.0.1:6876/3178977884,v1:127.0.0.1:6877/3178977884] exists,up 0253c251-b81a-4f43-af92-c81fb0549c27 2024-09-13T14:55:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:55:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:55:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:55:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:55:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:55:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:55:18.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:55:18.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:55:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:55:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:55:18.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:55:18.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:55:18.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:55:18.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:55:18.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:55:18.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:55:18.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:55:18.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:55:18.729 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:55:19.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:55:19.200 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:55:19.200 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:55:19.201 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:55:19.201 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:55:19.201 INFO:tasks.workunit.client.0.smithi112.stderr:5 2024-09-13T14:55:19.201 INFO:tasks.workunit.client.0.smithi112.stderr:6 2024-09-13T14:55:19.201 INFO:tasks.workunit.client.0.smithi112.stderr:7 2024-09-13T14:55:19.201 INFO:tasks.workunit.client.0.smithi112.stderr:8 2024-09-13T14:55:19.201 INFO:tasks.workunit.client.0.smithi112.stderr:9' 2024-09-13T14:55:19.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:55:19.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:19.201 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:55:19.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836572 2024-09-13T14:55:19.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836572 2024-09-13T14:55:19.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572' 2024-09-13T14:55:19.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:19.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:55:19.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673045 2024-09-13T14:55:19.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673045 2024-09-13T14:55:19.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673045' 2024-09-13T14:55:19.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:19.626 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:55:19.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509517 2024-09-13T14:55:19.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509517 2024-09-13T14:55:19.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509517' 2024-09-13T14:55:19.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:19.776 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:55:19.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645328 2024-09-13T14:55:19.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645328 2024-09-13T14:55:19.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509517 3-309237645328' 2024-09-13T14:55:19.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:19.920 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:55:20.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182465 2024-09-13T14:55:20.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182465 2024-09-13T14:55:20.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509517 3-309237645328 4-107374182465' 2024-09-13T14:55:20.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:20.064 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:55:20.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=356482285575 2024-09-13T14:55:20.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 356482285575 2024-09-13T14:55:20.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509517 3-309237645328 4-107374182465 5-356482285575' 2024-09-13T14:55:20.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:20.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:55:20.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855410 2024-09-13T14:55:20.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855410 2024-09-13T14:55:20.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509517 3-309237645328 4-107374182465 5-356482285575 6-150323855410' 2024-09-13T14:55:20.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:20.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:55:20.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691882 2024-09-13T14:55:20.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691882 2024-09-13T14:55:20.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509517 3-309237645328 4-107374182465 5-356482285575 6-150323855410 7-171798691882' 2024-09-13T14:55:20.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:20.508 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:55:20.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528359 2024-09-13T14:55:20.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528359 2024-09-13T14:55:20.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509517 3-309237645328 4-107374182465 5-356482285575 6-150323855410 7-171798691882 8-193273528359' 2024-09-13T14:55:20.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:20.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:55:20.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=382252089346 2024-09-13T14:55:20.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 382252089346 2024-09-13T14:55:20.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509517 3-309237645328 4-107374182465 5-356482285575 6-150323855410 7-171798691882 8-193273528359 9-382252089346' 2024-09-13T14:55:20.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:20.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836572 2024-09-13T14:55:20.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:20.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:55:20.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836572 2024-09-13T14:55:20.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:20.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836572 2024-09-13T14:55:20.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-09-13T14:55:20.826 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836572 2024-09-13T14:55:20.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:55:21.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836572 -lt 21474836572 2024-09-13T14:55:21.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:21.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673045 2024-09-13T14:55:21.149 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:21.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:55:21.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673045 2024-09-13T14:55:21.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:21.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673045 2024-09-13T14:55:21.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-09-13T14:55:21.153 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949673045 2024-09-13T14:55:21.154 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:55:21.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673045 -lt 42949673045 2024-09-13T14:55:21.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:21.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509517 2024-09-13T14:55:21.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:21.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:55:21.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509517 2024-09-13T14:55:21.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:21.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509517 2024-09-13T14:55:21.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509517' 2024-09-13T14:55:21.454 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509517 2024-09-13T14:55:21.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:55:21.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509517 -lt 64424509517 2024-09-13T14:55:21.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:21.781 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645328 2024-09-13T14:55:21.781 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:21.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:55:21.783 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645328 2024-09-13T14:55:21.783 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:21.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645328 2024-09-13T14:55:21.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645328' 2024-09-13T14:55:21.785 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.3 seq 309237645328 2024-09-13T14:55:21.785 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:55:22.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645328 -lt 309237645328 2024-09-13T14:55:22.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:22.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182465 2024-09-13T14:55:22.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:22.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:55:22.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182465 2024-09-13T14:55:22.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:22.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182465 2024-09-13T14:55:22.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182465' 2024-09-13T14:55:22.116 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.4 seq 107374182465 2024-09-13T14:55:22.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:55:22.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182465 -lt 107374182465 2024-09-13T14:55:22.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:22.458 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-356482285575 2024-09-13T14:55:22.458 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:22.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:55:22.461 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-356482285575 2024-09-13T14:55:22.461 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:22.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=356482285575 2024-09-13T14:55:22.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 356482285575' 2024-09-13T14:55:22.462 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.5 seq 356482285575 2024-09-13T14:55:22.462 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:55:22.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 356482285576 -lt 356482285575 2024-09-13T14:55:22.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:22.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855410 2024-09-13T14:55:22.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:22.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:55:22.816 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855410 2024-09-13T14:55:22.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:22.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855410 2024-09-13T14:55:22.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855410' 2024-09-13T14:55:22.822 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.6 seq 150323855410 2024-09-13T14:55:22.822 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:55:23.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855410 -lt 150323855410 2024-09-13T14:55:23.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:23.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691882 2024-09-13T14:55:23.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:23.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:55:23.150 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691882 2024-09-13T14:55:23.150 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:23.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691882 2024-09-13T14:55:23.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691882' 2024-09-13T14:55:23.152 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.7 seq 171798691882 2024-09-13T14:55:23.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:55:23.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691882 -lt 171798691882 2024-09-13T14:55:23.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:23.480 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528359 2024-09-13T14:55:23.480 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:23.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:55:23.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528359 2024-09-13T14:55:23.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:23.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528359 2024-09-13T14:55:23.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528359' 2024-09-13T14:55:23.486 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.8 seq 193273528359 2024-09-13T14:55:23.486 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:55:23.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528359 -lt 193273528359 2024-09-13T14:55:23.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:23.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-382252089346 2024-09-13T14:55:23.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:23.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:55:23.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-382252089346 2024-09-13T14:55:23.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:23.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=382252089346 2024-09-13T14:55:23.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 382252089346' 2024-09-13T14:55:23.819 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.9 seq 382252089346 2024-09-13T14:55:23.819 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:55:24.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 382252089346 -lt 382252089346 2024-09-13T14:55:24.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:55:24.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:55:24.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:55:24.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:55:24.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:55:24.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:55:24.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:55:24.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:55:24.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:55:24.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:55:24.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:55:24.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:55:24.940 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:55:24.941 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:55:24.941 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:55:25.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:55:25.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:55:25.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:55:25.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:55:25.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:55:25.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-09-13T14:55:25.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-13T14:55:25.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-13T14:55:25.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-09-13T14:55:25.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-09-13T14:55:25.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-13T14:55:25.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:55:25.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:55:25.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:55:25.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 259318"' 2024-09-13T14:55:25.371 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 259318' 2024-09-13T14:55:25.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-13T14:55:25.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:55:25.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:55:25.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 259319"' 2024-09-13T14:55:25.371 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 259319' 2024-09-13T14:55:25.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-13T14:55:25.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 259318 259319' 2024-09-13T14:55:25.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:55:25.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:55:25.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 259318 2024-09-13T14:55:25.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:55:25.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/259321: /' 2024-09-13T14:55:25.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-13T14:55:25.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/259323: /' 2024-09-13T14:55:29.789 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:55:29.789 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:55:29.789 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-13T14:55:29.789 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:55:29.790 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:55:29.790 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:55:29.790 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:55:29.790 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-13T14:55:29.790 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:55:29.790 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-13T14:55:29.790 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:55:29.790 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:55:29.790 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:55:29.790 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:55:29.790 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:55:29.790 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:55:29.791 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:55:29.791 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:55:29.791 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:55:29.791 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-13T14:55:29.791 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:55:29.791 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:55:29.791 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-13T14:55:29.791 INFO:tasks.workunit.client.0.smithi112.stderr:259321: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-13T14:55:29.791 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-13T14:55:29.791 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:55:29.791 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:55:29.791 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-13T14:55:29.792 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:55:29.792 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:55:29.792 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:55:29.792 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:55:29.794 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:55:29.794 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:55:29.794 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-13T14:55:29.794 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-13T14:55:29.794 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:55:29.795 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:55:29.795 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:55:29.795 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:55:29.795 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:55:29.795 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:55:29.795 INFO:tasks.workunit.client.0.smithi112.stderr:259321: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:55:29.795 INFO:tasks.workunit.client.0.smithi112.stderr:259321: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:55:29.795 INFO:tasks.workunit.client.0.smithi112.stderr:259321: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:55:29.795 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:55:29.795 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:55:29.795 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:55:29.795 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:55:29.796 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:55:29.796 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:55:29.796 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:55:29.796 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:55:29.796 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:55:29.796 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:55:29.796 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:55:29.796 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-13T14:55:29.796 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-13T14:55:29.796 INFO:tasks.workunit.client.0.smithi112.stderr:259321: start osd.5 2024-09-13T14:55:29.796 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:55:29.951 INFO:tasks.workunit.client.0.smithi112.stderr:2259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:55:29.951 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:55:29.951 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-13T14:55:29.951 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:55:29.951 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-09-13T14:55:29.951 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:55:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:55:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-13T14:55:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:55:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-13T14:55:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:55:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:55:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:55:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:55:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:55:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:55:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-09-13T14:55:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:55:29.952 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:55:29.953 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-13T14:55:29.953 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:55:29.953 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:55:29.953 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-09-13T14:55:29.953 INFO:tasks.workunit.client.0.smithi112.stderr:259323: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-13T14:55:29.953 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-13T14:55:29.953 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:55:29.953 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:55:29.953 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-13T14:55:29.953 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:55:29.953 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:55:29.953 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:55:29.953 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:55:29.955 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:55:29.955 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:55:29.955 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-13T14:55:29.955 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-13T14:55:29.955 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:55:29.955 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:55:29.955 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:55:29.955 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:55:29.955 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:55:29.955 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:55:29.955 INFO:tasks.workunit.client.0.smithi112.stderr:259323: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:55:29.956 INFO:tasks.workunit.client.0.smithi112.stderr:259323: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:55:29.956 INFO:tasks.workunit.client.0.smithi112.stderr:259323: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:55:29.956 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:55:29.956 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:55:29.956 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:55:29.956 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:55:29.956 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:55:29.956 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:55:29.956 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:55:29.956 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:55:29.956 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:55:29.956 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:55:29.956 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:55:29.957 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-13T14:55:29.957 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-13T14:55:29.957 INFO:tasks.workunit.client.0.smithi112.stderr:259323: start osd.9 2024-09-13T14:55:29.957 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:55:34.824 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-13T14:55:34.825 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-13T14:55:34.825 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:55:34.825 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:55:34.825 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:55:34.825 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 2024-09-13T14:55:29.827+0000 7fc2d23587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:55:34.825 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 2024-09-13T14:55:29.839+0000 7fc2d23587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:55:34.825 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 2024-09-13T14:55:29.843+0000 7fc2d23587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:55:34.825 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-13T14:55:34.825 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:55:34.825 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-13T14:55:34.825 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:55:34.825 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:55:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:55:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 0 2024-09-13T14:55:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:55:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:55:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:55:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:55:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 1 2024-09-13T14:55:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:55:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 2024-09-13T14:55:31.655+0000 7fc2d23587c0 -1 Falling back to public interface 2024-09-13T14:55:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:55:34.827 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:55:34.827 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:34.827 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:55:34.827 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 2 2024-09-13T14:55:34.827 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:34.827 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:55:34.827 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:55:34.827 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:55:34.827 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:34.827 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:55:34.827 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 3 2024-09-13T14:55:34.827 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:34.827 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:55:34.827 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:55:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 2024-09-13T14:55:34.823+0000 7fc2d23587c0 -59323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-13T14:55:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-13T14:55:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:55:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:55:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:55:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 2024-09-13T14:55:29.983+0000 7f052ef887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:55:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 2024-09-13T14:55:29.987+0000 7f052ef887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:55:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 2024-09-13T14:55:29.987+0000 7f052ef887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:55:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-13T14:55:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:55:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-13T14:55:34.878 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:55:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:55:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:55:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 0 2024-09-13T14:55:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:55:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:55:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 2024-09-13T14:55:31.335+0000 7f052ef887c0 -1 Falling back to public interface 2024-09-13T14:55:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:55:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:55:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 1 2024-09-13T14:55:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:55:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:55:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:55:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:55:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 2 2024-09-13T14:55:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:55:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:55:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 2024-09-13T14:55:34.231+0000 7f052ef887c0 -1 osd.9 91 log_to_monitors true 2024-09-13T14:55:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:55:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:55:34.880 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 3 2024-09-13T14:55:34.881 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:55:34.881 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:37.367 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:55:37.368 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:55:37.368 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:37.368 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:55:37.368 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 4 2024-09-13T14:55:37.368 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:37.368 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:55:37.368 INFO:tasks.workunit.client.0.smithi112.stderr:259323: osd.9 up in weight 1 up_from 96 up_thru 0 down_at 92 last_clean_interval [89,91) [v2:127.0.0.1:6842/2749721325,v1:127.0.0.1:6843/2749721325] [v2:127.0.0.1:6844/2749721325,v1:127.0.0.1:6845/2749721325] exists,up 0253c251-b81a-4f43-af92-c81fb0549c27 2024-09-13T14:55:37.368 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:55:37.368 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:55:37.368 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:55:37.369 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:55:37.369 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:55:37.369 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:55:37.369 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:55:37.369 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:55:37.369 INFO:tasks.workunit.client.0.smithi112.stderr:259323: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:55:37.372 INFO:tasks.workunit.client.0.smithi112.stderr:259323: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:55:37.372 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:55:37.372 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:55:37.372 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:55:37.372 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:55:37.372 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:55:37.372 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:55:37.372 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:55:37.372 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:55:37.372 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:55:37.372 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:55:37.373 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 1 2024-09-13T14:55:37.373 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 2 2024-09-13T14:55:37.373 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 3 2024-09-13T14:55:37.373 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 4 2024-09-13T14:55:37.373 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 5 2024-09-13T14:55:37.373 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 6 2024-09-13T14:55:37.373 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 7 2024-09-13T14:55:37.373 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 8 2024-09-13T14:55:37.373 INFO:tasks.workunit.client.0.smithi112.stderr:259323: 9' 2024-09-13T14:55:37.373 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:55:37.373 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:37.373 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:55:37.373 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836577 2024-09-13T14:55:37.374 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836577 2024-09-13T14:55:37.374 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577' 2024-09-13T14:55:37.374 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:38.055 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //1 osd.5 91 log_to_monitors true 2024-09-13T14:55:38.055 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:55:38.055 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:38.055 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:55:38.055 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 4 2024-09-13T14:55:38.055 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:38.055 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:55:38.055 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:55:38.055 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:55:38.055 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:55:38.055 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:55:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 5 2024-09-13T14:55:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:55:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:55:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:259321: osd.5 up in weight 1 up_from 99 up_thru 99 down_at 92 last_clean_interval [83,91) [v2:127.0.0.1:6874/264732825,v1:127.0.0.1:6875/264732825] [v2:127.0.0.1:6876/264732825,v1:127.0.0.1:6877/264732825] exists,up 3df67c99-00a1-4d12-b468-6f9b9faac1b1 2024-09-13T14:55:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:55:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:55:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:55:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:55:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:55:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:55:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:55:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:55:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:259321: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:55:38.056 INFO:tasks.workunit.client.0.smithi112.stderr:259321: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:55:38.057 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:55:38.057 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:55:38.057 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:55:38.057 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:55:38.057 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:55:38.057 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:55:38.057 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:55:38.057 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:55:38.057 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:55:38.057 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:55:38.057 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 1 2024-09-13T14:55:38.057 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 2 2024-09-13T14:55:38.057 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 3 2024-09-13T14:55:38.058 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 4 2024-09-13T14:55:38.058 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 5 2024-09-13T14:55:38.058 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 6 2024-09-13T14:55:38.058 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 7 2024-09-13T14:55:38.058 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 8 2024-09-13T14:55:38.058 INFO:tasks.workunit.client.0.smithi112.stderr:259321: 9' 2024-09-13T14:55:38.058 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:55:38.938 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shhome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:55:38.938 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673050 2024-09-13T14:55:38.938 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673050 2024-09-13T14:55:38.938 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050' 2024-09-13T14:55:38.938 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:38.938 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:55:38.939 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509522 2024-09-13T14:55:38.939 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509522 2024-09-13T14:55:38.939 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509522' 2024-09-13T14:55:38.939 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:38.939 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:55:38.939 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645333 2024-09-13T14:55:38.939 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645333 2024-09-13T14:55:38.940 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509522 3-309237645333' 2024-09-13T14:55:38.940 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:38.940 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:55:38.940 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182470 2024-09-13T14:55:38.940 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182470 2024-09-13T14:55:38.940 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509522 3-309237645333 4-107374182470' 2024-09-13T14:55:38.941 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:38.941 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:55:38.941 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=425201762306 2024-09-13T14:55:38.941 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 425201762306 2024-09-13T14:55:38.941 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509522 3-309237645333 4-107374182470 5-425201762306' 2024-09-13T14:55:38.941 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:38.941 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:55:38.941 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855415 2024-09-13T14:55:38.941 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855415 2024-09-13T14:55:38.941 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509522 3-309237645333 4-107374182470 5-425201762306 6-150323855415' 2024-09-13T14:55:38.942 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:38.942 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:55:39.373 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_p:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:39.373 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:55:39.374 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836578 2024-09-13T14:55:39.374 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836578 2024-09-13T14:55:39.374 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578' 2024-09-13T14:55:39.374 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:39.374 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:55:39.374 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673051 2024-09-13T14:55:39.374 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673051 2024-09-13T14:55:39.374 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673051' 2024-09-13T14:55:39.374 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:39.374 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:55:39.374 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509523 2024-09-13T14:55:39.374 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509523 2024-09-13T14:55:39.374 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509523' 2024-09-13T14:55:39.375 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:39.375 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:55:39.375 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645334 2024-09-13T14:55:39.375 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645334 2024-09-13T14:55:39.375 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509523 3-309237645334' 2024-09-13T14:55:39.375 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:39.375 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:55:39.375 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182471 2024-09-13T14:55:39.376 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182471 2024-09-13T14:55:39.376 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509523 3-309237645334 4-107374182471' 2024-09-13T14:55:39.376 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:39.376 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:55:39.376 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=425201762307 2024-09-13T14:55:39.376 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 425201762307 2024-09-13T14:55:39.376 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509523 3-309237645334 4-107374182471 5-425201762307' 2024-09-13T14:55:39.376 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:39.376 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:55:39.376 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855416 2024-09-13T14:55:39.532 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /homeg_stats: seq=171798691887 2024-09-13T14:55:39.532 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691887 2024-09-13T14:55:39.532 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509522 3-309237645333 4-107374182470 5-425201762306 6-150323855415 7-171798691887' 2024-09-13T14:55:39.533 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:39.533 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:55:39.533 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528364 2024-09-13T14:55:39.533 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528364 2024-09-13T14:55:39.533 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509522 3-309237645333 4-107374182470 5-425201762306 6-150323855415 7-171798691887 8-193273528364' 2024-09-13T14:55:39.533 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:39.533 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:55:39.533 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860418 2024-09-13T14:55:39.534 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860418 2024-09-13T14:55:39.534 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673050 2-64424509522 3-309237645333 4-107374182470 5-425201762306 6-150323855415 7-171798691887 8-193273528364 9-412316860418' 2024-09-13T14:55:39.534 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:39.534 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836577 2024-09-13T14:55:39.534 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:39.534 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:55:39.534 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836577 2024-09-13T14:55:39.534 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:39.534 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836577 2024-09-13T14:55:39.534 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-09-13T14:55:39.535 INFO:tasks.workunit.client.0.smithi112.stderr:259323: waiting osd.0 seq 21474836577 2024-09-13T14:55:39.535 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:55:39.535 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836577 -lt 21474836577 2024-09-13T14:55:39.535 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:39.535 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673050 2024-09-13T14:55:39.536 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:39.536 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:55:39.536 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673050 2024-09-13T14:55:39.536 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:39.536 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673050 2024-09-13T14:55:39.536 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-09-13T14:55:39.536 INFO:tasks.workunit.client.0.smithi112.stderr:259323: waiting osd.1 seq 42949673050 2024-09-13T14:55:41.038 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:55:41.038 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673050 -lt 42949673050 2024-09-13T14:55:41.038 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:41.038 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:41.038 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509522 2024-09-13T14:55:41.038 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:55:41.038 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509522 2024-09-13T14:55:41.038 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:41.038 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509522 2024-09-13T14:55:41.039 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509522' 2024-09-13T14:55:41.039 INFO:tasks.workunit.client.0.smithi112.stderr:259323: waiting osd.2 seq 64424509522 2024-09-13T14:55:41.039 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:55:41.039 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509522 -lt 64424509522 2024-09-13T14:55:41.039 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:41.039 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645333 2024-09-13T14:55:41.039 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:41.039 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:55:41.039 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645333 2024-09-13T14:55:41.039 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:41.039 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645333 2024-09-13T14:55:41.039 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645333' 2024-09-13T14:55:41.039 INFO:tasks.workunit.client.0.smithi112.stderr:259323: waiting osd.3 seq 309237645333 2024-09-13T14:55:41.039 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:55:41.040 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645334 -lt 309237645333 2024-09-13T14:55:41.040 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:41.040 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:41.040 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182470 2024-09-13T14:55:41.040 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:55:41.040 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182470 2024-09-13T14:55:41.040 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:41.040 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182470 2024-09-13T14:55:41.040 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182470' 2024-09-13T14:55:41.040 INFO:tasks.workunit.client.0.smithi112.stderr:259323: waiting osd.4 seq 107374182470 2024-09-13T14:55:41.040 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:55:41.040 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182471 -lt 107374182470 2024-09-13T14:55:41.040 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:41.041 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-425201762306 2024-09-13T14:55:41.709 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cep/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855416 2024-09-13T14:55:41.709 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509523 3-309237645334 4-107374182471 5-425201762307 6-150323855416' 2024-09-13T14:55:41.709 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:41.709 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:55:41.710 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691888 2024-09-13T14:55:41.710 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691888 2024-09-13T14:55:41.710 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509523 3-309237645334 4-107374182471 5-425201762307 6-150323855416 7-171798691888' 2024-09-13T14:55:41.710 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:41.710 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:55:41.710 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528365 2024-09-13T14:55:41.710 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528365 2024-09-13T14:55:41.710 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509523 3-309237645334 4-107374182471 5-425201762307 6-150323855416 7-171798691888 8-193273528365' 2024-09-13T14:55:41.710 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:55:41.710 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:55:41.710 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860420 2024-09-13T14:55:41.710 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860420 2024-09-13T14:55:41.710 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509523 3-309237645334 4-107374182471 5-425201762307 6-150323855416 7-171798691888 8-193273528365 9-412316860420' 2024-09-13T14:55:41.711 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:41.711 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836578 2024-09-13T14:55:41.711 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:41.711 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:55:41.711 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836578 2024-09-13T14:55:41.711 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:41.711 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836578 2024-09-13T14:55:41.711 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2024-09-13T14:55:41.711 INFO:tasks.workunit.client.0.smithi112.stderr:259321: waiting osd.0 seq 21474836578 2024-09-13T14:55:41.711 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:55:41.711 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836577 -lt 21474836578 2024-09-13T14:55:41.711 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:55:41.712 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:55:41.712 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:55:42.055 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helhtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:42.055 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:55:42.055 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-425201762306 2024-09-13T14:55:42.055 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:42.055 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=425201762306 2024-09-13T14:55:42.055 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 425201762306' 2024-09-13T14:55:42.056 INFO:tasks.workunit.client.0.smithi112.stderr:259323: waiting osd.5 seq 425201762306 2024-09-13T14:55:42.056 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:55:42.056 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 425201762307 -lt 425201762306 2024-09-13T14:55:42.056 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:42.056 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855415 2024-09-13T14:55:42.056 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:42.056 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:55:42.056 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855415 2024-09-13T14:55:42.056 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:42.056 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855415 2024-09-13T14:55:42.057 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855415' 2024-09-13T14:55:42.057 INFO:tasks.workunit.client.0.smithi112.stderr:259323: waiting osd.6 seq 150323855415 2024-09-13T14:55:42.057 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:55:42.057 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855416 -lt 150323855415 2024-09-13T14:55:42.057 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:42.057 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691887 2024-09-13T14:55:42.057 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:42.057 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:55:42.057 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691887 2024-09-13T14:55:42.057 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:42.057 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691887 2024-09-13T14:55:42.057 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691887' 2024-09-13T14:55:42.058 INFO:tasks.workunit.client.0.smithi112.stderr:259323: waiting osd.7 seq 171798691887 2024-09-13T14:55:42.058 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:55:42.058 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691888 -lt 171798691887 2024-09-13T14:55:42.058 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:42.058 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528364 2024-09-13T14:55:42.058 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:42.058 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:55:42.058 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528364 2024-09-13T14:55:42.729 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cupers.sh:2222: flush_pg_stats: test 21474836578 -lt 21474836578 2024-09-13T14:55:42.729 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:42.729 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673051 2024-09-13T14:55:42.729 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:42.729 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:55:42.729 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673051 2024-09-13T14:55:42.729 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:42.729 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673051 2024-09-13T14:55:42.729 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-09-13T14:55:42.729 INFO:tasks.workunit.client.0.smithi112.stderr:259321: waiting osd.1 seq 42949673051 2024-09-13T14:55:42.730 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:55:42.730 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673051 -lt 42949673051 2024-09-13T14:55:42.730 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:42.730 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509523 2024-09-13T14:55:42.730 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:42.730 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:55:42.730 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509523 2024-09-13T14:55:42.730 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:42.730 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509523 2024-09-13T14:55:42.730 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509523' 2024-09-13T14:55:42.730 INFO:tasks.workunit.client.0.smithi112.stderr:259321: waiting osd.2 seq 64424509523 2024-09-13T14:55:42.730 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:55:42.730 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509524 -lt 64424509523 2024-09-13T14:55:42.731 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:42.731 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645334 2024-09-13T14:55:42.731 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:42.731 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:55:42.731 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645334 2024-09-13T14:55:42.731 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:42.731 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645334 2024-09-13T14:55:42.731 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645334' 2024-09-13T14:55:42.731 INFO:tasks.workunit.client.0.smithi112.stderr:259321: waiting osd.3 seq 309237645334 2024-09-13T14:55:42.731 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:55:42.731 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645335 -lt 309237645334 2024-09-13T14:55:42.731 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:42.731 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182471 2024-09-13T14:55:42.731 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:43.719 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:55:43.719 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182471 2024-09-13T14:55:43.719 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:43.719 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182471 2024-09-13T14:55:43.720 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182471' 2024-09-13T14:55:43.720 INFO:tasks.workunit.client.0.smithi112.stderr:259321: waiting osd.4 seq 107374182471 2024-09-13T14:55:43.720 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:55:43.720 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182471 -lt 107374182471 2024-09-13T14:55:43.720 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:43.720 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-425201762307 2024-09-13T14:55:43.720 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:43.720 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:55:43.720 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-425201762307 2024-09-13T14:55:43.720 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:43.720 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=425201762307 2024-09-13T14:55:43.720 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 425201762307' 2024-09-13T14:55:43.720 INFO:tasks.workunit.client.0.smithi112.stderr:259321: waiting osd.5 seq 425201762307 2024-09-13T14:55:43.720 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:55:43.721 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 425201762308 -lt 425201762307 2024-09-13T14:55:43.721 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:43.721 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855416 2024-09-13T14:55:43.721 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:43.721 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:55:43.721 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855416 2024-09-13T14:55:43.721 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:43.721 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855416 2024-09-13T14:55:43.721 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855416' 2024-09-13T14:55:43.721 INFO:tasks.workunit.client.0.smithi112.stderr:259321: waiting osd.6 seq 150323855416 2024-09-13T14:55:43.721 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:55:43.721 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855416 -lt 150323855416 2024-09-13T14:55:43.721 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:43.722 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691888 2024-09-13T14:55:43.722 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:43.722 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:55:43.722 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691888 2024-09-13T14:55:43.722 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:43.818 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stat -d - -f 2 2024-09-13T14:55:43.818 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528364 2024-09-13T14:55:43.818 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528364' 2024-09-13T14:55:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:259323: waiting osd.8 seq 193273528364 2024-09-13T14:55:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:55:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528365 -lt 193273528364 2024-09-13T14:55:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-412316860418 2024-09-13T14:55:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:55:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-412316860418 2024-09-13T14:55:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860418 2024-09-13T14:55:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 412316860418' 2024-09-13T14:55:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:259323: waiting osd.9 seq 412316860418 2024-09-13T14:55:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:55:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860420 -lt 412316860418 2024-09-13T14:55:43.820 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:55:43.820 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:55:43.820 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:55:43.820 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:55:43.820 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:55:43.820 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:55:43.820 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:55:43.820 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:55:43.820 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:55:43.820 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:55:43.820 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:55:43.820 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:55:43.821 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:55:43.821 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:55:43.821 INFO:tasks.workunit.client.0.smithi112.stderr:259323: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:55:43.821 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:55:43.821 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:55:43.821 INFO:tasks.workunit.client.0.smithi112.stderr:259323: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:55:43.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:55:45.051 INFO:tasks.workunit.client.0.smithi112.stderr:ts: seq=171798691888 2024-09-13T14:55:45.051 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691888' 2024-09-13T14:55:45.051 INFO:tasks.workunit.client.0.smithi112.stderr:259321: waiting osd.7 seq 171798691888 2024-09-13T14:55:45.051 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:55:45.051 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691888 -lt 171798691888 2024-09-13T14:55:45.051 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:45.051 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:45.051 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528365 2024-09-13T14:55:45.051 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:55:45.051 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528365 2024-09-13T14:55:45.051 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:45.051 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528365 2024-09-13T14:55:45.052 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528365' 2024-09-13T14:55:45.052 INFO:tasks.workunit.client.0.smithi112.stderr:259321: waiting osd.8 seq 193273528365 2024-09-13T14:55:45.052 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:55:45.052 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528366 -lt 193273528365 2024-09-13T14:55:45.052 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:55:45.052 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-412316860420 2024-09-13T14:55:45.052 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:55:45.052 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:55:45.052 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-412316860420 2024-09-13T14:55:45.052 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:55:45.052 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860420 2024-09-13T14:55:45.052 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 412316860420' 2024-09-13T14:55:45.052 INFO:tasks.workunit.client.0.smithi112.stderr:259321: waiting osd.9 seq 412316860420 2024-09-13T14:55:45.053 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:55:45.053 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860420 -lt 412316860420 2024-09-13T14:55:45.053 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:55:45.053 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:55:45.053 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:55:45.053 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:55:45.053 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:55:45.053 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:55:45.053 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:55:45.053 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:55:45.053 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:55:45.784 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:55:45.784 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:55:45.785 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:55:45.785 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:55:45.785 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:55:45.785 INFO:tasks.workunit.client.0.smithi112.stderr:259321: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:55:45.785 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:55:45.785 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:55:45.785 INFO:tasks.workunit.client.0.smithi112.stderr:259321: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:55:45.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:55:45.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:55:45.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 259319 2024-09-13T14:55:45.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:55:45.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:55:45.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:55:45.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-13T14:55:45.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:55:45.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-13T14:55:45.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:55:45.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:55:45.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:55:45.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:55:46.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-13T14:55:46.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-13T14:55:46.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:55:46.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:55:46.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:55:46.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:55:46.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:55:46.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:55:46.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:55:06.106012+0000 2024-09-13T14:55:46.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:55:46.767 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:55:46.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:55:06.106012+0000 2024-09-13T14:55:46.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:55:46.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:55:06.106012+0000 2024-09-13T14:55:46.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:55:46.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:55:46.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:55:46.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:55:46.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:55:46.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:55:46.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:55:46.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:55:47.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:55:06.106012+0000 '>' 2024-09-13T14:55:06.106012+0000 2024-09-13T14:55:47.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:55:48.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:55:48.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:55:48.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:55:48.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:55:48.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:55:48.119 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:55:48.119 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:55:48.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:55:06.106012+0000 '>' 2024-09-13T14:55:06.106012+0000 2024-09-13T14:55:48.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:55:49.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:55:49.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:55:49.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:55:49.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:55:49.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:55:49.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:55:49.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:55:49.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:55:06.106012+0000 '>' 2024-09-13T14:55:06.106012+0000 2024-09-13T14:55:49.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:55:50.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:55:50.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:55:50.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:55:50.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:55:50.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:55:50.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:55:50.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:55:51.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:55:06.106012+0000 '>' 2024-09-13T14:55:06.106012+0000 2024-09-13T14:55:51.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:55:52.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:55:52.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:55:52.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:55:52.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:55:52.167 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:55:52.167 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:55:52.167 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:55:52.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:55:47.400853+0000 '>' 2024-09-13T14:55:06.106012+0000 2024-09-13T14:55:52.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:55:52.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:55:54.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-13T14:55:54.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:55:54.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:55:54.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:55:54.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 262842"' 2024-09-13T14:55:54.504 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 262842' 2024-09-13T14:55:54.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:55:54.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:55:54.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:55:54.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 262844"' 2024-09-13T14:55:54.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 262844' 2024-09-13T14:55:54.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:55:54.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-13T14:55:54.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 262842 262844' 2024-09-13T14:55:54.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:55:54.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/262845: /' 2024-09-13T14:55:54.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:55:54.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 262842 2024-09-13T14:55:54.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:55:54.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/262847: /' 2024-09-13T14:55:55.688 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:55:55.688 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:55:55.688 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-13T14:55:55.688 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:55:55.689 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:55:55.689 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:55:55.689 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:55:55.689 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-13T14:55:55.689 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:55:55.689 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-13T14:55:55.690 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:55:55.690 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:55:55.690 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:55:55.690 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:55:55.690 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:55:55.690 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:55:55.691 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-13T14:55:55.691 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:55:55.691 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:55:55.691 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-13T14:55:55.691 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:55:55.691 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:55:55.691 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-13T14:55:55.691 INFO:tasks.workunit.client.0.smithi112.stderr:262847: _ 2024-09-13T14:55:55.692 INFO:tasks.workunit.client.0.smithi112.stderr:262847: hinfo_key 2024-09-13T14:55:55.692 INFO:tasks.workunit.client.0.smithi112.stderr:262847: snapset 2024-09-13T14:55:55.692 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-13T14:55:55.692 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:55:55.692 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:55:55.692 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-13T14:55:55.692 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:55:55.693 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-13T14:55:55.693 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:55:55.693 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:55:55.693 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:55:55.694 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:55:55.694 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-13T14:55:55.694 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-13T14:55:55.694 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:55:55.694 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:55:55.694 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:55:55.694 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:55:55.694 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:55:55.694 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:55:55.695 INFO:tasks.workunit.client.0.smithi112.stderr:262847: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:55:55.695 INFO:tasks.workunit.client.0.smithi112.stderr:262847: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:55:55.695 INFO:tasks.workunit.client.0.smithi112.stderr:262847: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:55:55.695 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:55:55.695 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:55:55.695 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:55:55.695 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:55:55.695 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:55:55.695 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:55:55.695 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:55:55.695 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:55:55.695 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:55:55.695 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:55:55.695 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:55:55.696 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-13T14:55:55.696 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-13T14:55:55.696 INFO:tasks.workunit.client.0.smithi112.stderr:262847: start osd.9 2024-09-13T14:55:55.732 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:55:55.732 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:55:55.733 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-13T14:55:55.733 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:55:55.733 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:55:55.733 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:55:55.733 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:55:55.733 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-13T14:55:55.734 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:55:55.734 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-13T14:55:55.734 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:55:55.734 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:55:55.734 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:55:55.735 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:55:55.735 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:55:55.735 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:55:55.735 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:55:55.735 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:55:55.735 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:55:55.736 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-13T14:55:55.736 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:55:55.736 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:55:55.736 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-13T14:55:55.736 INFO:tasks.workunit.client.0.smithi112.stderr:262845: _ 2024-09-13T14:55:55.736 INFO:tasks.workunit.client.0.smithi112.stderr:262845: hinfo_key 2024-09-13T14:55:55.737 INFO:tasks.workunit.client.0.smithi112.stderr:262845: snapset 2024-09-13T14:55:55.737 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-13T14:55:55.737 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:55:55.737 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:55:55.737 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-13T14:55:55.738 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:55:55.738 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:55:55.738 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:55:55.738 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:55:55.739 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:55:55.739 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:55:55.739 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-13T14:55:55.739 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-13T14:55:55.739 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:55:55.740 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:55:55.740 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:55:55.740 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:55:55.740 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:55:55.740 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:55:55.740 INFO:tasks.workunit.client.0.smithi112.stderr:262845: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:55:55.741 INFO:tasks.workunit.client.0.smithi112.stderr:262845: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:55:55.741 INFO:tasks.workunit.client.0.smithi112.stderr:262845: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:55:55.741 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:55:55.741 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:55:55.741 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:55:55.742 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:55:55.742 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:55:55.742 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:55:55.742 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:55:55.742 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:55:55.742 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:55:55.743 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:55:55.743 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:55:55.743 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-13T14:55:55.743 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-13T14:55:55.743 INFO:tasks.workunit.client.0.smithi112.stderr:262845: start osd.5 2024-09-13T14:56:00.535 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:56:00.535 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-13T14:56:00.535 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-13T14:56:00.535 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:56:00.535 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:56:00.535 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:56:00.535 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 2024-09-13T14:55:55.726+0000 7f5f0cbdc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:00.535 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 2024-09-13T14:55:55.834+0000 7f5f0cbdc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:00.535 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 2024-09-13T14:55:55.838+0000 7f5f0cbdc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:00.535 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-13T14:56:00.536 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:56:00.536 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-13T14:56:00.536 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:56:00.536 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:56:00.536 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:00.536 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:56:00.536 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 0 2024-09-13T14:56:00.536 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:00.536 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:56:00.536 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:00.536 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 2024-09-13T14:55:56.890+0000 7f5f0cbdc7c0 -1 Falling back to public interface 2024-09-13T14:56:00.536 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:56:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 1 2024-09-13T14:56:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:56:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:56:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 2 2024-09-13T14:56:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:56:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:00.538 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:00.538 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:00.538 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:56:00.538 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 3 2024-09-13T14:56:00.538 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:00.538 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:56:00.538 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:01.160 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 2024-09-13T14:56:00.534+0000 7f5fvery_ops 2024-09-13T14:56:01.161 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-13T14:56:01.161 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-13T14:56:01.161 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:56:01.161 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:56:01.161 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:56:01.161 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 2024-09-13T14:55:55.762+0000 7f0679c197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:01.161 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 2024-09-13T14:55:55.834+0000 7f0679c197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:01.161 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 2024-09-13T14:55:55.838+0000 7f0679c197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:01.161 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-13T14:56:01.161 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:56:01.161 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-13T14:56:01.161 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:56:01.162 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:56:01.162 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:01.162 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:56:01.162 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 0 2024-09-13T14:56:01.162 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:01.162 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:01.162 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:01.162 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 2024-09-13T14:55:57.370+0000 7f0679c197c0 -1 Falling back to public interface 2024-09-13T14:56:01.162 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:01.162 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:01.162 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:56:01.162 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 1 2024-09-13T14:56:01.162 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:01.162 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:01.162 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:01.163 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:01.163 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:01.163 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:56:01.163 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 2 2024-09-13T14:56:01.163 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:01.163 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:01.163 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:01.163 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:01.163 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:01.163 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:56:01.163 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 3 2024-09-13T14:56:01.163 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:01.163 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:01.164 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:02.579 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 2024-09-13T14:56:01.158+0000 7f060cbdc7c0 -1 osd.9 101 log_to_monitors true 2024-09-13T14:56:02.579 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:02.579 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:02.579 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:56:02.579 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 4 2024-09-13T14:56:02.579 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:02.579 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-13T14:56:02.579 INFO:tasks.workunit.client.0.smithi112.stderr:262847: osd.9 up in weight 1 up_from 105 up_thru 0 down_at 102 last_clean_interval [96,101) [v2:127.0.0.1:6842/297656574,v1:127.0.0.1:6843/297656574] [v2:127.0.0.1:6844/297656574,v1:127.0.0.1:6845/297656574] exists,up 0253c251-b81a-4f43-af92-c81fb0549c27 2024-09-13T14:56:02.579 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:56:02.579 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:56:02.579 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:56:02.579 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:56:02.579 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:56:02.580 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:56:02.580 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:56:02.580 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:56:02.580 INFO:tasks.workunit.client.0.smithi112.stderr:262847: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:56:02.580 INFO:tasks.workunit.client.0.smithi112.stderr:262847: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:56:02.580 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:56:02.580 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:56:02.580 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:56:02.580 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:56:02.580 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:56:02.580 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:56:02.580 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:56:02.580 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:56:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:56:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:56:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 1 2024-09-13T14:56:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 2 2024-09-13T14:56:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 3 2024-09-13T14:56:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 4 2024-09-13T14:56:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 5 2024-09-13T14:56:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 6 2024-09-13T14:56:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 7 2024-09-13T14:56:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 8 2024-09-13T14:56:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:262847: 9' 2024-09-13T14:56:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:56:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:02.582 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:56:02.582 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836584 2024-09-13T14:56:02.582 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836584 2024-09-13T14:56:02.582 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584' 2024-09-13T14:56:02.582 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:03.605 INFO:tasks.workunit.client.0.smithi112.stderr:79c197c0 -1 osd.5 102 log_to_monitors true 2024-09-13T14:56:03.606 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:03.606 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:03.606 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:56:03.606 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 4 2024-09-13T14:56:03.606 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:03.606 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:03.606 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:03.606 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:03.606 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:03.606 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:56:03.606 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 5 2024-09-13T14:56:03.606 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:03.606 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:03.607 INFO:tasks.workunit.client.0.smithi112.stderr:262845: osd.5 up in weight 1 up_from 106 up_thru 106 down_at 102 last_clean_interval [99,102) [v2:127.0.0.1:6874/1727602077,v1:127.0.0.1:6875/1727602077] [v2:127.0.0.1:6876/1727602077,v1:127.0.0.1:6877/1727602077] exists,up 3df67c99-00a1-4d12-b468-6f9b9faac1b1 2024-09-13T14:56:03.607 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:56:03.607 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:56:03.607 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:56:03.607 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:56:03.607 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:56:03.607 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:56:03.607 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:56:03.607 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:56:03.607 INFO:tasks.workunit.client.0.smithi112.stderr:262845: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:56:03.607 INFO:tasks.workunit.client.0.smithi112.stderr:262845: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:56:03.607 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:56:03.607 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:56:03.607 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:56:03.608 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:56:03.608 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:56:03.608 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:56:03.608 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:56:03.608 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:56:03.608 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:56:03.608 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:56:03.608 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 1 2024-09-13T14:56:03.608 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 2 2024-09-13T14:56:03.608 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 3 2024-09-13T14:56:03.608 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 4 2024-09-13T14:56:03.608 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 5 2024-09-13T14:56:03.608 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 6 2024-09-13T14:56:03.609 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 7 2024-09-13T14:56:03.609 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 8 2024-09-13T14:56:03.609 INFO:tasks.workunit.client.0.smithi112.stderr:262845: 9' 2024-09-13T14:56:03.609 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:56:04.182 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standal262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:56:04.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673057 2024-09-13T14:56:04.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673057 2024-09-13T14:56:04.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673057' 2024-09-13T14:56:04.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:04.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:56:04.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509529 2024-09-13T14:56:04.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509529 2024-09-13T14:56:04.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509529' 2024-09-13T14:56:04.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:04.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:56:04.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645340 2024-09-13T14:56:04.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645340 2024-09-13T14:56:04.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509529 3-309237645340' 2024-09-13T14:56:04.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:04.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:56:04.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182477 2024-09-13T14:56:04.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182477 2024-09-13T14:56:04.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509529 3-309237645340 4-107374182477' 2024-09-13T14:56:04.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:04.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:56:04.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533378 2024-09-13T14:56:04.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533378 2024-09-13T14:56:04.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509529 3-309237645340 4-107374182477 5-455266533378' 2024-09-13T14:56:04.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:04.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:56:04.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855422 2024-09-13T14:56:04.185 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855422 2024-09-13T14:56:04.185 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509529 3-309237645340 4-107374182477 5-455266533378 6-150323855422' 2024-09-13T14:56:04.185 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:04.185 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:56:04.722 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221one/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:04.722 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:56:04.722 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836585 2024-09-13T14:56:04.722 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836585 2024-09-13T14:56:04.723 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585' 2024-09-13T14:56:04.723 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:04.723 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:56:04.723 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673058 2024-09-13T14:56:04.723 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673058 2024-09-13T14:56:04.723 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673058' 2024-09-13T14:56:04.723 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:04.723 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:56:04.723 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509530 2024-09-13T14:56:04.723 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509530 2024-09-13T14:56:04.723 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509530' 2024-09-13T14:56:04.723 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:04.723 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:56:04.724 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645341 2024-09-13T14:56:04.724 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645341 2024-09-13T14:56:04.724 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509530 3-309237645341' 2024-09-13T14:56:04.724 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:04.724 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:56:04.724 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182478 2024-09-13T14:56:04.724 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182478 2024-09-13T14:56:04.724 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509530 3-309237645341 4-107374182478' 2024-09-13T14:56:04.724 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:04.724 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:56:04.724 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533379 2024-09-13T14:56:04.724 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533379 2024-09-13T14:56:04.724 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509530 3-309237645341 4-107374182478 5-455266533379' 2024-09-13T14:56:04.724 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:04.725 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:56:04.770 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=15032380: flush_pg_stats: seq=171798691894 2024-09-13T14:56:04.770 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691894 2024-09-13T14:56:04.770 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509529 3-309237645340 4-107374182477 5-455266533378 6-150323855422 7-171798691894' 2024-09-13T14:56:04.770 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:04.770 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:56:04.770 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528371 2024-09-13T14:56:04.770 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528371 2024-09-13T14:56:04.770 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509529 3-309237645340 4-107374182477 5-455266533378 6-150323855422 7-171798691894 8-193273528371' 2024-09-13T14:56:04.770 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:04.771 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:56:04.771 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566082 2024-09-13T14:56:04.771 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566082 2024-09-13T14:56:04.771 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509529 3-309237645340 4-107374182477 5-455266533378 6-150323855422 7-171798691894 8-193273528371 9-450971566082' 2024-09-13T14:56:04.771 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:04.771 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836584 2024-09-13T14:56:04.771 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:04.771 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:56:04.771 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836584 2024-09-13T14:56:04.771 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:04.771 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836584 2024-09-13T14:56:04.771 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-09-13T14:56:04.771 INFO:tasks.workunit.client.0.smithi112.stderr:262847: waiting osd.0 seq 21474836584 2024-09-13T14:56:04.772 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:56:04.772 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836585 -lt 21474836584 2024-09-13T14:56:04.772 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:04.772 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673057 2024-09-13T14:56:04.772 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:04.772 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:56:04.772 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673057 2024-09-13T14:56:04.772 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:04.772 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673057 2024-09-13T14:56:04.772 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-09-13T14:56:04.772 INFO:tasks.workunit.client.0.smithi112.stderr:262847: waiting osd.1 seq 42949673057 2024-09-13T14:56:05.445 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: f55423 2024-09-13T14:56:05.445 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855423 2024-09-13T14:56:05.445 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509530 3-309237645341 4-107374182478 5-455266533379 6-150323855423' 2024-09-13T14:56:05.445 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:05.445 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:56:05.445 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691895 2024-09-13T14:56:05.445 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691895 2024-09-13T14:56:05.445 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509530 3-309237645341 4-107374182478 5-455266533379 6-150323855423 7-171798691895' 2024-09-13T14:56:05.446 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:05.446 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:56:05.446 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528372 2024-09-13T14:56:05.446 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528372 2024-09-13T14:56:05.446 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509530 3-309237645341 4-107374182478 5-455266533379 6-150323855423 7-171798691895 8-193273528372' 2024-09-13T14:56:05.446 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:05.446 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:56:05.446 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566083 2024-09-13T14:56:05.446 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566083 2024-09-13T14:56:05.446 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509530 3-309237645341 4-107374182478 5-455266533379 6-150323855423 7-171798691895 8-193273528372 9-450971566083' 2024-09-13T14:56:05.446 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:05.446 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836585 2024-09-13T14:56:05.446 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:05.446 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:56:05.447 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836585 2024-09-13T14:56:05.447 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:05.447 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836585 2024-09-13T14:56:05.447 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-09-13T14:56:05.447 INFO:tasks.workunit.client.0.smithi112.stderr:262845: waiting osd.0 seq 21474836585 2024-09-13T14:56:05.447 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:56:05.447 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836585 -lt 21474836585 2024-09-13T14:56:05.447 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:05.447 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673058 2024-09-13T14:56:05.447 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:06.181 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/stlush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:56:06.181 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673058 -lt 42949673057 2024-09-13T14:56:06.181 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:06.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509529 2024-09-13T14:56:06.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:06.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:56:06.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509529 2024-09-13T14:56:06.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:06.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509529 2024-09-13T14:56:06.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-09-13T14:56:06.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: waiting osd.2 seq 64424509529 2024-09-13T14:56:06.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:56:06.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509529 -lt 64424509529 2024-09-13T14:56:06.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:06.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:06.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645340 2024-09-13T14:56:06.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:56:06.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645340 2024-09-13T14:56:06.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:06.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645340 2024-09-13T14:56:06.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645340' 2024-09-13T14:56:06.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: waiting osd.3 seq 309237645340 2024-09-13T14:56:06.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:56:06.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645340 -lt 309237645340 2024-09-13T14:56:06.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:06.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182477 2024-09-13T14:56:06.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:06.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:56:06.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182477 2024-09-13T14:56:06.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:06.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182477 2024-09-13T14:56:06.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182477' 2024-09-13T14:56:06.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: waiting osd.4 seq 107374182477 2024-09-13T14:56:06.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:56:06.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182477 -lt 107374182477 2024-09-13T14:56:06.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:06.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-455266533378 2024-09-13T14:56:07.181 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:07.181 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:56:07.181 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-455266533378 2024-09-13T14:56:07.181 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:07.181 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533378 2024-09-13T14:56:07.181 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 455266533378' 2024-09-13T14:56:07.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: waiting osd.5 seq 455266533378 2024-09-13T14:56:07.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:56:07.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533379 -lt 455266533378 2024-09-13T14:56:07.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:07.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855422 2024-09-13T14:56:07.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:07.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:56:07.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855422 2024-09-13T14:56:07.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:07.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855422 2024-09-13T14:56:07.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855422' 2024-09-13T14:56:07.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: waiting osd.6 seq 150323855422 2024-09-13T14:56:07.182 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:56:07.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855423 -lt 150323855422 2024-09-13T14:56:07.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:07.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691894 2024-09-13T14:56:07.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:07.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:56:07.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691894 2024-09-13T14:56:07.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:07.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691894 2024-09-13T14:56:07.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691894' 2024-09-13T14:56:07.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: waiting osd.7 seq 171798691894 2024-09-13T14:56:07.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:56:07.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691895 -lt 171798691894 2024-09-13T14:56:07.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:07.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:07.183 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528371 2024-09-13T14:56:07.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:56:07.184 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528371 2024-09-13T14:56:07.803 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_andalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:56:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673058 2024-09-13T14:56:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673058 2024-09-13T14:56:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673058' 2024-09-13T14:56:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:262845: waiting osd.1 seq 42949673058 2024-09-13T14:56:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:56:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673058 -lt 42949673058 2024-09-13T14:56:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509530 2024-09-13T14:56:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:56:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509530 2024-09-13T14:56:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:07.805 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509530 2024-09-13T14:56:07.805 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-09-13T14:56:07.805 INFO:tasks.workunit.client.0.smithi112.stderr:262845: waiting osd.2 seq 64424509530 2024-09-13T14:56:07.805 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:56:07.805 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509529 -lt 64424509530 2024-09-13T14:56:07.805 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:56:07.805 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:56:07.805 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:56:07.805 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509531 -lt 64424509530 2024-09-13T14:56:07.805 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:07.805 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645341 2024-09-13T14:56:07.805 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:07.805 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:56:07.805 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645341 2024-09-13T14:56:07.806 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:07.806 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645341 2024-09-13T14:56:07.806 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645341' 2024-09-13T14:56:07.806 INFO:tasks.workunit.client.0.smithi112.stderr:262845: waiting osd.3 seq 309237645341 2024-09-13T14:56:07.806 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:56:07.806 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645341 -lt 309237645341 2024-09-13T14:56:07.806 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:07.806 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182478 2024-09-13T14:56:08.870 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:08.871 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:56:08.871 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182478 2024-09-13T14:56:08.871 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:08.871 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182478 2024-09-13T14:56:08.871 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182478' 2024-09-13T14:56:08.871 INFO:tasks.workunit.client.0.smithi112.stderr:262845: waiting osd.4 seq 107374182478 2024-09-13T14:56:08.871 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:56:08.871 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182478 -lt 107374182478 2024-09-13T14:56:08.871 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:08.871 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-455266533379 2024-09-13T14:56:08.871 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:08.871 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:56:08.872 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-455266533379 2024-09-13T14:56:08.872 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:08.872 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533379 2024-09-13T14:56:08.872 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 455266533379' 2024-09-13T14:56:08.872 INFO:tasks.workunit.client.0.smithi112.stderr:262845: waiting osd.5 seq 455266533379 2024-09-13T14:56:08.872 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:56:08.872 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533380 -lt 455266533379 2024-09-13T14:56:08.872 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:08.872 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855423 2024-09-13T14:56:08.872 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:08.872 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:56:08.872 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855423 2024-09-13T14:56:08.873 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:08.873 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855423 2024-09-13T14:56:08.873 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855423' 2024-09-13T14:56:08.876 INFO:tasks.workunit.client.0.smithi112.stderr:262845: waiting osd.6 seq 150323855423 2024-09-13T14:56:08.876 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:56:08.876 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855424 -lt 150323855423 2024-09-13T14:56:08.876 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:08.877 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:08.877 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691895 2024-09-13T14:56:08.877 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:56:08.877 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691895 2024-09-13T14:56:08.877 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:09.024 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shstats: cut -d - -f 2 2024-09-13T14:56:09.025 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528371 2024-09-13T14:56:09.025 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528371' 2024-09-13T14:56:09.025 INFO:tasks.workunit.client.0.smithi112.stderr:262847: waiting osd.8 seq 193273528371 2024-09-13T14:56:09.025 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:56:09.025 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528372 -lt 193273528371 2024-09-13T14:56:09.025 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:09.025 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566082 2024-09-13T14:56:09.025 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:09.025 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:56:09.025 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566082 2024-09-13T14:56:09.025 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:09.025 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566082 2024-09-13T14:56:09.026 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566082' 2024-09-13T14:56:09.026 INFO:tasks.workunit.client.0.smithi112.stderr:262847: waiting osd.9 seq 450971566082 2024-09-13T14:56:09.026 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:56:09.026 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566084 -lt 450971566082 2024-09-13T14:56:09.026 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:56:09.026 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:56:09.026 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:56:09.026 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:56:09.026 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:56:09.026 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:56:09.026 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:56:09.026 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:56:09.026 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:56:09.027 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:56:09.027 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:56:09.027 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:56:09.027 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:56:09.027 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:56:09.027 INFO:tasks.workunit.client.0.smithi112.stderr:262847: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:56:09.027 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:56:09.027 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:56:09.027 INFO:tasks.workunit.client.0.smithi112.stderr:262847: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:56:09.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:56:10.251 INFO:tasks.workunit.client.0.smithi112.stderr::2220: flush_pg_stats: seq=171798691895 2024-09-13T14:56:10.251 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691895' 2024-09-13T14:56:10.251 INFO:tasks.workunit.client.0.smithi112.stderr:262845: waiting osd.7 seq 171798691895 2024-09-13T14:56:10.251 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:56:10.251 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691896 -lt 171798691895 2024-09-13T14:56:10.251 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:10.251 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528372 2024-09-13T14:56:10.251 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:10.251 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:56:10.251 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528372 2024-09-13T14:56:10.251 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:10.252 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528372 2024-09-13T14:56:10.252 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528372' 2024-09-13T14:56:10.252 INFO:tasks.workunit.client.0.smithi112.stderr:262845: waiting osd.8 seq 193273528372 2024-09-13T14:56:10.252 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:56:10.252 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528372 -lt 193273528372 2024-09-13T14:56:10.252 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:10.252 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566083 2024-09-13T14:56:10.252 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:10.252 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:56:10.252 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566083 2024-09-13T14:56:10.252 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:10.252 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566083 2024-09-13T14:56:10.253 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566083' 2024-09-13T14:56:10.253 INFO:tasks.workunit.client.0.smithi112.stderr:262845: waiting osd.9 seq 450971566083 2024-09-13T14:56:10.253 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:56:10.253 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566084 -lt 450971566083 2024-09-13T14:56:10.253 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:56:10.253 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:56:10.253 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:56:10.253 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:56:10.253 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:56:10.253 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:56:10.253 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:56:10.253 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:56:10.253 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:56:10.942 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:56:10.942 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:56:10.942 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:56:10.942 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:56:10.943 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:56:10.943 INFO:tasks.workunit.client.0.smithi112.stderr:262845: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:56:10.943 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:56:10.943 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:56:10.943 INFO:tasks.workunit.client.0.smithi112.stderr:262845: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:56:10.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:56:10.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:56:10.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 262844 2024-09-13T14:56:10.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:56:10.943 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:56:10.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:56:10.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-13T14:56:10.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:56:10.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:56:10.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:56:10.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-09-13T14:56:10.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-13T14:56:10.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-13T14:56:10.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-13T14:56:10.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-09-13T14:56:10.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-13T14:56:10.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:56:10.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:56:10.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:56:10.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 265824"' 2024-09-13T14:56:10.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 265824' 2024-09-13T14:56:10.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:56:10.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:56:10.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:56:10.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 265825"' 2024-09-13T14:56:10.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 265825' 2024-09-13T14:56:10.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-13T14:56:10.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 265824 265825' 2024-09-13T14:56:10.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:56:10.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:56:10.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 265824 2024-09-13T14:56:10.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:56:10.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/265827: /' 2024-09-13T14:56:10.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:56:10.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/265829: /' 2024-09-13T14:56:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:56:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:56:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:56:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:56:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:56:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:56:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:56:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:56:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:56:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:56:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:56:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:56:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:56:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:56:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:56:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:56:14.926 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T14:56:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:56:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:56:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:56:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:56:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:56:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-13T14:56:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:265827: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-13T14:56:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:56:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:56:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:56:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:56:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:56:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:56:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:56:14.927 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:56:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:56:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:56:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:56:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:56:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:56:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:56:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:56:14.932 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:56:14.932 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:56:14.932 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:56:14.932 INFO:tasks.workunit.client.0.smithi112.stderr:265827: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:56:14.932 INFO:tasks.workunit.client.0.smithi112.stderr:265827: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:56:14.932 INFO:tasks.workunit.client.0.smithi112.stderr:265827: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:56:14.932 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:56:14.932 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:56:14.932 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:56:14.932 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:56:14.932 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:56:14.932 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:56:14.932 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:56:14.933 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:56:14.933 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:56:14.933 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:56:14.933 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:56:14.933 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:56:14.933 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:56:14.933 INFO:tasks.workunit.client.0.smithi112.stderr:265827: start osd.3 2024-09-13T14:56:14.933 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:56:15.352 INFO:tasks.workunit.client.0.smithi112.stderr:2265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:56:15.352 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:56:15.352 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-13T14:56:15.352 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:56:15.352 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:56:15.352 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:56:15.352 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:56:15.352 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-13T14:56:15.352 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:56:15.353 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-13T14:56:15.353 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:56:15.353 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:56:15.353 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:56:15.353 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:56:15.353 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:56:15.353 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:56:15.353 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-13T14:56:15.353 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:56:15.353 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:56:15.353 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-13T14:56:15.353 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:56:15.353 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:56:15.354 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-13T14:56:15.354 INFO:tasks.workunit.client.0.smithi112.stderr:265829: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-13T14:56:15.354 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-13T14:56:15.354 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:56:15.354 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:56:15.354 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-13T14:56:15.354 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:56:15.354 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:56:15.354 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:56:15.354 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:56:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:56:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:56:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-13T14:56:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-13T14:56:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:56:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:56:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:56:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:56:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:56:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:56:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:265829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:56:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:265829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:56:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:265829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:56:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:56:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:56:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:56:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:56:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:56:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:56:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:56:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:56:15.358 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:56:15.358 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:56:15.358 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:56:15.358 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-13T14:56:15.358 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-13T14:56:15.358 INFO:tasks.workunit.client.0.smithi112.stderr:265829: start osd.5 2024-09-13T14:56:15.358 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:56:19.660 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:56:19.661 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:56:19.661 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:56:19.661 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:56:19.661 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:56:19.661 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 2024-09-13T14:56:14.970+0000 7fba5fb6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:19.662 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 2024-09-13T14:56:14.970+0000 7fba5fb6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:19.662 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 2024-09-13T14:56:14.974+0000 7fba5fb6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:19.662 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:56:19.662 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:56:19.662 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:56:19.663 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:56:19.663 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:56:19.663 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:19.663 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:56:19.663 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 0 2024-09-13T14:56:19.663 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:19.664 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:56:19.664 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:19.664 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 2024-09-13T14:56:16.266+0000 7fba5fb6a7c0 -1 Falling back to public interface 2024-09-13T14:56:19.664 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:19.664 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:19.665 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:56:19.665 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 1 2024-09-13T14:56:19.665 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:19.665 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:56:19.665 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:19.666 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:19.666 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:19.666 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:56:19.666 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 2 2024-09-13T14:56:19.666 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:19.666 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:56:19.667 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:19.667 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 2024-09-13T14:56:19.153+0000 7fba5fb6a7c0 -1 osd.3 108 log_to_monitors true 2024-09-13T14:56:19.667 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:19.667 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:19.667 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:56:19.668 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 3 2024-09-13T14:56:19.668 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:19.668 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:56:20.457 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper65829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-13T14:56:20.458 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-13T14:56:20.458 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:56:20.458 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:56:20.458 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:56:20.458 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 2024-09-13T14:56:15.386+0000 7f70e36467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:20.458 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 2024-09-13T14:56:15.390+0000 7f70e36467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:20.458 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 2024-09-13T14:56:15.394+0000 7f70e36467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:20.458 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-13T14:56:20.458 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:56:20.458 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-13T14:56:20.458 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:56:20.458 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:56:20.459 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:20.459 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:56:20.459 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 0 2024-09-13T14:56:20.459 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:20.459 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:20.459 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:20.459 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 2024-09-13T14:56:16.666+0000 7f70e36467c0 -1 Falling back to public interface 2024-09-13T14:56:20.459 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:20.459 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:20.459 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:56:20.459 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 1 2024-09-13T14:56:20.459 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:20.460 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:20.460 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:20.460 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:20.460 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:20.460 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:56:20.460 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 2 2024-09-13T14:56:20.460 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:20.460 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:20.460 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:20.460 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 2024-09-13T14:56:19.693+0000 7f70e36467c0 -1 osd.5 108 log_to_monitors true 2024-09-13T14:56:20.460 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:20.460 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:20.461 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:56:20.461 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 3 2024-09-13T14:56:20.461 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:20.461 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:21.940 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:21.941 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:21.941 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:21.941 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:56:21.941 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 4 2024-09-13T14:56:21.941 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:21.941 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:56:21.941 INFO:tasks.workunit.client.0.smithi112.stderr:265827: osd.3 up in weight 1 up_from 113 up_thru 113 down_at 109 last_clean_interval [72,108) [v2:127.0.0.1:6826/1336026150,v1:127.0.0.1:6827/1336026150] [v2:127.0.0.1:6828/1336026150,v1:127.0.0.1:6829/1336026150] exists,up 858e19ad-bbe8-4eeb-8da5-f6f455b0cb32 2024-09-13T14:56:21.942 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:56:21.942 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:56:21.942 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:56:21.942 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:56:21.942 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:56:21.942 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:56:21.942 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:56:21.942 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:56:21.942 INFO:tasks.workunit.client.0.smithi112.stderr:265827: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:56:21.942 INFO:tasks.workunit.client.0.smithi112.stderr:265827: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:56:21.942 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:56:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:56:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:56:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:56:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:56:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:56:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:56:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:56:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:56:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:56:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 1 2024-09-13T14:56:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 2 2024-09-13T14:56:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 3 2024-09-13T14:56:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 4 2024-09-13T14:56:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 5 2024-09-13T14:56:21.944 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 6 2024-09-13T14:56:21.944 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 7 2024-09-13T14:56:21.944 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 8 2024-09-13T14:56:21.944 INFO:tasks.workunit.client.0.smithi112.stderr:265827: 9' 2024-09-13T14:56:21.944 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:56:21.944 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:21.944 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:56:21.944 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836590 2024-09-13T14:56:21.944 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836590 2024-09-13T14:56:21.944 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590' 2024-09-13T14:56:21.944 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:22.706 INFO:tasks.workunit.client.0.smithi112.stderr:2658s.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:22.706 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:22.707 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:22.707 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:56:22.707 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 4 2024-09-13T14:56:22.707 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:22.707 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:22.707 INFO:tasks.workunit.client.0.smithi112.stderr:265829: osd.5 up in weight 1 up_from 115 up_thru 115 down_at 109 last_clean_interval [106,108) [v2:127.0.0.1:6874/2493896279,v1:127.0.0.1:6875/2493896279] [v2:127.0.0.1:6876/2493896279,v1:127.0.0.1:6877/2493896279] exists,up 3df67c99-00a1-4d12-b468-6f9b9faac1b1 2024-09-13T14:56:22.707 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:56:22.707 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:56:22.707 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:56:22.707 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:56:22.707 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:56:22.707 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:56:22.707 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:56:22.707 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:56:22.708 INFO:tasks.workunit.client.0.smithi112.stderr:265829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:56:22.708 INFO:tasks.workunit.client.0.smithi112.stderr:265829: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:56:22.708 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:56:22.708 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:56:22.708 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:56:22.708 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:56:22.708 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:56:22.708 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:56:22.708 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:56:22.708 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:56:22.708 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:56:22.708 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:56:22.708 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 1 2024-09-13T14:56:22.709 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 2 2024-09-13T14:56:22.709 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 3 2024-09-13T14:56:22.709 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 4 2024-09-13T14:56:22.709 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 5 2024-09-13T14:56:22.709 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 6 2024-09-13T14:56:22.709 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 7 2024-09-13T14:56:22.709 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 8 2024-09-13T14:56:22.709 INFO:tasks.workunit.client.0.smithi112.stderr:265829: 9' 2024-09-13T14:56:22.709 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:56:22.709 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:22.709 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:56:22.709 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836591 2024-09-13T14:56:22.710 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836591 2024-09-13T14:56:22.710 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591' 2024-09-13T14:56:22.710 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:23.428 INFO:tasks.workunit.client.0.smithi112.stderr:26527: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:56:23.429 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673063 2024-09-13T14:56:23.429 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673063 2024-09-13T14:56:23.429 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673063' 2024-09-13T14:56:23.429 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:23.429 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:56:23.429 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509535 2024-09-13T14:56:23.429 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509535 2024-09-13T14:56:23.429 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535' 2024-09-13T14:56:23.429 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:23.429 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:56:23.429 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=485331304450 2024-09-13T14:56:23.430 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 485331304450 2024-09-13T14:56:23.430 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535 3-485331304450' 2024-09-13T14:56:23.430 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:23.430 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:56:23.430 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182483 2024-09-13T14:56:23.430 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182483 2024-09-13T14:56:23.430 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535 3-485331304450 4-107374182483' 2024-09-13T14:56:23.430 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:23.430 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:56:23.430 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=493921239042 2024-09-13T14:56:23.430 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 493921239042 2024-09-13T14:56:23.430 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535 3-485331304450 4-107374182483 5-493921239042' 2024-09-13T14:56:23.430 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:23.430 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:56:23.431 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855428 2024-09-13T14:56:23.431 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855428 2024-09-13T14:56:23.431 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535 3-485331304450 4-107374182483 5-493921239042 6-150323855428' 2024-09-13T14:56:23.431 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:23.431 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:56:23.831 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: f829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:56:23.831 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673064 2024-09-13T14:56:23.832 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673064 2024-09-13T14:56:23.832 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673064' 2024-09-13T14:56:23.832 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:23.832 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:56:23.832 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509536 2024-09-13T14:56:23.832 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509536 2024-09-13T14:56:23.832 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509536' 2024-09-13T14:56:23.832 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:23.832 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:56:23.832 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=485331304451 2024-09-13T14:56:23.832 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 485331304451 2024-09-13T14:56:23.832 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509536 3-485331304451' 2024-09-13T14:56:23.833 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:23.833 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:56:23.833 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182484 2024-09-13T14:56:23.833 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182484 2024-09-13T14:56:23.833 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509536 3-485331304451 4-107374182484' 2024-09-13T14:56:23.833 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:23.833 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:56:23.833 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=493921239043 2024-09-13T14:56:23.833 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 493921239043 2024-09-13T14:56:23.833 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509536 3-485331304451 4-107374182484 5-493921239043' 2024-09-13T14:56:23.833 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:23.833 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:56:23.833 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855429 2024-09-13T14:56:23.834 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855429 2024-09-13T14:56:23.834 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509536 3-485331304451 4-107374182484 5-493921239043 6-150323855429' 2024-09-13T14:56:23.834 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:23.834 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:56:24.011 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: lush_pg_stats: seq=171798691900 2024-09-13T14:56:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691900 2024-09-13T14:56:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535 3-485331304450 4-107374182483 5-493921239042 6-150323855428 7-171798691900' 2024-09-13T14:56:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:56:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528376 2024-09-13T14:56:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528376 2024-09-13T14:56:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535 3-485331304450 4-107374182483 5-493921239042 6-150323855428 7-171798691900 8-193273528376' 2024-09-13T14:56:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:56:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566088 2024-09-13T14:56:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566088 2024-09-13T14:56:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673063 2-64424509535 3-485331304450 4-107374182483 5-493921239042 6-150323855428 7-171798691900 8-193273528376 9-450971566088' 2024-09-13T14:56:24.012 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:24.013 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836590 2024-09-13T14:56:24.013 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:24.013 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:56:24.013 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836590 2024-09-13T14:56:24.013 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:24.013 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836590 2024-09-13T14:56:24.013 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836590' 2024-09-13T14:56:24.013 INFO:tasks.workunit.client.0.smithi112.stderr:265827: waiting osd.0 seq 21474836590 2024-09-13T14:56:24.013 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:56:24.013 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836590 -lt 21474836590 2024-09-13T14:56:24.013 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:24.013 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673063 2024-09-13T14:56:24.013 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:24.013 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:56:24.014 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673063 2024-09-13T14:56:24.014 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:24.014 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673063 2024-09-13T14:56:24.014 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2024-09-13T14:56:24.014 INFO:tasks.workunit.client.0.smithi112.stderr:265827: waiting osd.1 seq 42949673063 2024-09-13T14:56:24.556 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flushflush_pg_stats: seq=171798691901 2024-09-13T14:56:24.556 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691901 2024-09-13T14:56:24.556 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509536 3-485331304451 4-107374182484 5-493921239043 6-150323855429 7-171798691901' 2024-09-13T14:56:24.556 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:24.556 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:56:24.556 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528378 2024-09-13T14:56:24.557 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528378 2024-09-13T14:56:24.557 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509536 3-485331304451 4-107374182484 5-493921239043 6-150323855429 7-171798691901 8-193273528378' 2024-09-13T14:56:24.557 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:24.557 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:56:24.557 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566089 2024-09-13T14:56:24.557 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566089 2024-09-13T14:56:24.557 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509536 3-485331304451 4-107374182484 5-493921239043 6-150323855429 7-171798691901 8-193273528378 9-450971566089' 2024-09-13T14:56:24.557 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:24.557 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836591 2024-09-13T14:56:24.557 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:24.558 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:56:24.558 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836591 2024-09-13T14:56:24.558 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:24.558 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836591 2024-09-13T14:56:24.558 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2024-09-13T14:56:24.558 INFO:tasks.workunit.client.0.smithi112.stderr:265829: waiting osd.0 seq 21474836591 2024-09-13T14:56:24.558 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:56:24.558 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836591 -lt 21474836591 2024-09-13T14:56:24.558 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:24.558 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673064 2024-09-13T14:56:24.558 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:24.558 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:56:24.558 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673064 2024-09-13T14:56:24.558 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:24.559 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673064 2024-09-13T14:56:24.559 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673064' 2024-09-13T14:56:24.559 INFO:tasks.workunit.client.0.smithi112.stderr:265829: waiting osd.1 seq 42949673064 2024-09-13T14:56:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flus_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:56:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673064 -lt 42949673063 2024-09-13T14:56:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509535 2024-09-13T14:56:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:56:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509535 2024-09-13T14:56:25.407 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:25.408 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509535 2024-09-13T14:56:25.408 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509535' 2024-09-13T14:56:25.408 INFO:tasks.workunit.client.0.smithi112.stderr:265827: waiting osd.2 seq 64424509535 2024-09-13T14:56:25.408 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:56:25.408 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509536 -lt 64424509535 2024-09-13T14:56:25.408 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:25.408 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-485331304450 2024-09-13T14:56:25.408 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:25.408 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:56:25.408 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-485331304450 2024-09-13T14:56:25.408 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:25.408 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=485331304450 2024-09-13T14:56:25.408 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 485331304450' 2024-09-13T14:56:25.408 INFO:tasks.workunit.client.0.smithi112.stderr:265827: waiting osd.3 seq 485331304450 2024-09-13T14:56:25.409 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:56:25.409 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 485331304451 -lt 485331304450 2024-09-13T14:56:25.409 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:25.409 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182483 2024-09-13T14:56:25.409 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:25.409 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:56:25.409 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182483 2024-09-13T14:56:25.409 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:25.409 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182483 2024-09-13T14:56:25.409 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182483' 2024-09-13T14:56:25.409 INFO:tasks.workunit.client.0.smithi112.stderr:265827: waiting osd.4 seq 107374182483 2024-09-13T14:56:25.409 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:56:25.409 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182484 -lt 107374182483 2024-09-13T14:56:25.409 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:25.410 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-493921239042 2024-09-13T14:56:25.930 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubunh_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:56:25.930 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673064 -lt 42949673064 2024-09-13T14:56:25.930 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:25.930 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509536 2024-09-13T14:56:25.930 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:25.930 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:56:25.930 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509536 2024-09-13T14:56:25.930 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:25.930 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509536 2024-09-13T14:56:25.931 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-09-13T14:56:25.931 INFO:tasks.workunit.client.0.smithi112.stderr:265829: waiting osd.2 seq 64424509536 2024-09-13T14:56:25.931 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:56:25.931 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509536 -lt 64424509536 2024-09-13T14:56:25.931 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:25.931 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-485331304451 2024-09-13T14:56:25.931 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:25.931 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:56:25.931 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-485331304451 2024-09-13T14:56:25.931 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:25.931 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=485331304451 2024-09-13T14:56:25.931 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 485331304451' 2024-09-13T14:56:25.931 INFO:tasks.workunit.client.0.smithi112.stderr:265829: waiting osd.3 seq 485331304451 2024-09-13T14:56:25.932 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:56:25.932 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 485331304451 -lt 485331304451 2024-09-13T14:56:25.932 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:25.932 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:25.932 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182484 2024-09-13T14:56:25.932 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:56:25.932 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182484 2024-09-13T14:56:25.932 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:25.932 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182484 2024-09-13T14:56:25.932 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182484' 2024-09-13T14:56:25.932 INFO:tasks.workunit.client.0.smithi112.stderr:265829: waiting osd.4 seq 107374182484 2024-09-13T14:56:25.932 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:56:25.932 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182484 -lt 107374182484 2024-09-13T14:56:25.932 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:25.933 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-493921239043 2024-09-13T14:56:26.394 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubutu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:26.394 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:56:26.394 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-493921239042 2024-09-13T14:56:26.394 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:26.394 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=493921239042 2024-09-13T14:56:26.395 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 493921239042' 2024-09-13T14:56:26.395 INFO:tasks.workunit.client.0.smithi112.stderr:265827: waiting osd.5 seq 493921239042 2024-09-13T14:56:26.395 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:56:26.395 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 493921239043 -lt 493921239042 2024-09-13T14:56:26.395 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:26.395 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855428 2024-09-13T14:56:26.395 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:26.395 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:56:26.395 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855428 2024-09-13T14:56:26.395 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:26.395 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855428 2024-09-13T14:56:26.395 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855428' 2024-09-13T14:56:26.395 INFO:tasks.workunit.client.0.smithi112.stderr:265827: waiting osd.6 seq 150323855428 2024-09-13T14:56:26.395 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:56:26.395 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855429 -lt 150323855428 2024-09-13T14:56:26.396 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:26.396 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691900 2024-09-13T14:56:26.396 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:26.396 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:56:26.396 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691900 2024-09-13T14:56:26.396 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:26.396 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691900 2024-09-13T14:56:26.396 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-09-13T14:56:26.396 INFO:tasks.workunit.client.0.smithi112.stderr:265827: waiting osd.7 seq 171798691900 2024-09-13T14:56:26.396 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:56:26.396 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691901 -lt 171798691900 2024-09-13T14:56:26.396 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:26.396 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528376 2024-09-13T14:56:26.397 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:26.397 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:56:26.397 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528376 2024-09-13T14:56:26.898 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_statntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:26.898 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:56:26.898 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-493921239043 2024-09-13T14:56:26.898 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:26.898 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=493921239043 2024-09-13T14:56:26.899 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 493921239043' 2024-09-13T14:56:26.899 INFO:tasks.workunit.client.0.smithi112.stderr:265829: waiting osd.5 seq 493921239043 2024-09-13T14:56:26.899 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:56:26.899 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 493921239043 -lt 493921239043 2024-09-13T14:56:26.899 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:26.899 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855429 2024-09-13T14:56:26.899 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:26.899 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:56:26.899 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855429 2024-09-13T14:56:26.899 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:26.899 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855429 2024-09-13T14:56:26.899 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855429' 2024-09-13T14:56:26.899 INFO:tasks.workunit.client.0.smithi112.stderr:265829: waiting osd.6 seq 150323855429 2024-09-13T14:56:26.899 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:56:26.900 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855429 -lt 150323855429 2024-09-13T14:56:26.900 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:26.900 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691901 2024-09-13T14:56:26.900 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:26.900 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:56:26.900 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691901 2024-09-13T14:56:26.900 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:26.900 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691901 2024-09-13T14:56:26.900 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691901' 2024-09-13T14:56:26.900 INFO:tasks.workunit.client.0.smithi112.stderr:265829: waiting osd.7 seq 171798691901 2024-09-13T14:56:26.900 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:56:26.900 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691901 -lt 171798691901 2024-09-13T14:56:26.900 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:26.901 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528378 2024-09-13T14:56:26.901 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:26.901 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:56:26.901 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528378 2024-09-13T14:56:28.130 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stas: cut -d - -f 2 2024-09-13T14:56:28.131 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528376 2024-09-13T14:56:28.131 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528376' 2024-09-13T14:56:28.131 INFO:tasks.workunit.client.0.smithi112.stderr:265827: waiting osd.8 seq 193273528376 2024-09-13T14:56:28.131 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:56:28.131 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528378 -lt 193273528376 2024-09-13T14:56:28.131 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:28.131 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566088 2024-09-13T14:56:28.131 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:28.131 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:56:28.131 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566088 2024-09-13T14:56:28.131 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:28.132 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566088 2024-09-13T14:56:28.132 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566088' 2024-09-13T14:56:28.132 INFO:tasks.workunit.client.0.smithi112.stderr:265827: waiting osd.9 seq 450971566088 2024-09-13T14:56:28.132 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:56:28.132 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566090 -lt 450971566088 2024-09-13T14:56:28.132 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:56:28.132 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:56:28.132 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:56:28.132 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:56:28.132 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:56:28.132 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:56:28.132 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:56:28.132 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:56:28.133 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:56:28.133 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:56:28.133 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:56:28.133 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:56:28.133 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:56:28.133 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:56:28.133 INFO:tasks.workunit.client.0.smithi112.stderr:265827: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:56:28.133 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:56:28.133 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:56:28.133 INFO:tasks.workunit.client.0.smithi112.stderr:265827: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:56:28.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:56:28.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:56:28.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 265825 2024-09-13T14:56:28.649 INFO:tasks.workunit.client.0.smithi112.stderr:ts: cut -d - -f 2 2024-09-13T14:56:28.649 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528378 2024-09-13T14:56:28.649 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528378' 2024-09-13T14:56:28.649 INFO:tasks.workunit.client.0.smithi112.stderr:265829: waiting osd.8 seq 193273528378 2024-09-13T14:56:28.650 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:56:28.650 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528378 -lt 193273528378 2024-09-13T14:56:28.650 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:28.650 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:28.650 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566089 2024-09-13T14:56:28.650 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:56:28.650 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566089 2024-09-13T14:56:28.650 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:28.650 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566089 2024-09-13T14:56:28.650 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566089' 2024-09-13T14:56:28.650 INFO:tasks.workunit.client.0.smithi112.stderr:265829: waiting osd.9 seq 450971566089 2024-09-13T14:56:28.650 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:56:28.650 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566090 -lt 450971566089 2024-09-13T14:56:28.650 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:56:28.650 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:56:28.651 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:56:28.651 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:56:28.651 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:56:28.651 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:56:28.651 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:56:28.651 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:56:28.651 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:56:28.651 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:56:28.651 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:56:28.651 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:56:28.651 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:56:28.651 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:56:28.651 INFO:tasks.workunit.client.0.smithi112.stderr:265829: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:56:28.652 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:56:28.652 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:56:28.652 INFO:tasks.workunit.client.0.smithi112.stderr:265829: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:56:28.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:56:28.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:56:28.652 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:56:28.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:56:28.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-13T14:56:28.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:56:28.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-13T14:56:28.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T14:56:28.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:56:28.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:56:28.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T14:56:29.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-13T14:56:29.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-13T14:56:29.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T14:56:29.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T14:56:29.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:56:29.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:56:29.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:56:29.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:56:29.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:55:47.400853+0000 2024-09-13T14:56:29.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T14:56:29.605 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-13T14:56:29.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T14:55:47.400853+0000 2024-09-13T14:56:29.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T14:56:29.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:55:47.400853+0000 2024-09-13T14:56:29.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:56:29.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:56:29.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:56:29.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:56:29.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:56:29.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:56:29.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:56:29.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:56:29.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:55:47.400853+0000 '>' 2024-09-13T14:55:47.400853+0000 2024-09-13T14:56:29.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:56:30.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:56:30.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:56:30.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:56:30.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:56:30.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:56:30.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:56:30.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:56:31.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:55:47.400853+0000 '>' 2024-09-13T14:55:47.400853+0000 2024-09-13T14:56:31.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:56:32.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:56:32.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:56:32.249 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:56:32.249 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:56:32.249 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:56:32.250 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:56:32.250 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:56:32.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:55:47.400853+0000 '>' 2024-09-13T14:55:47.400853+0000 2024-09-13T14:56:32.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:56:33.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:56:33.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:56:33.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:56:33.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:56:33.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:56:33.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:56:33.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:56:33.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:55:47.400853+0000 '>' 2024-09-13T14:55:47.400853+0000 2024-09-13T14:56:33.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:56:34.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:56:34.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:56:34.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:56:34.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:56:34.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:56:34.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:56:34.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:56:35.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:55:47.400853+0000 '>' 2024-09-13T14:55:47.400853+0000 2024-09-13T14:56:35.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:56:36.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:56:36.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:56:36.249 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T14:56:36.249 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T14:56:36.249 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:56:36.249 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:56:36.250 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T14:56:36.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:56:30.269083+0000 '>' 2024-09-13T14:55:47.400853+0000 2024-09-13T14:56:36.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:56:36.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:56:38.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-13T14:56:38.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:56:38.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:56:38.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:56:38.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 269326"' 2024-09-13T14:56:38.583 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 269326' 2024-09-13T14:56:38.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:56:38.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T14:56:38.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T14:56:38.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 269328"' 2024-09-13T14:56:38.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 269328' 2024-09-13T14:56:38.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/269329: /' 2024-09-13T14:56:38.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:56:38.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-13T14:56:38.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 269326 269328' 2024-09-13T14:56:38.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T14:56:38.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:56:38.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 269326 2024-09-13T14:56:38.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:56:38.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/269331: /' 2024-09-13T14:56:39.737 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:56:39.737 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:56:39.737 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-13T14:56:39.737 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:56:39.737 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:56:39.737 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:56:39.738 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:56:39.738 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-13T14:56:39.738 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:56:39.738 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-13T14:56:39.738 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:56:39.738 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:56:39.738 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:56:39.738 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:56:39.738 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:56:39.738 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:56:39.738 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-13T14:56:39.739 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:56:39.739 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:56:39.739 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-13T14:56:39.739 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:56:39.739 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:56:39.739 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-13T14:56:39.739 INFO:tasks.workunit.client.0.smithi112.stderr:269331: _ 2024-09-13T14:56:39.739 INFO:tasks.workunit.client.0.smithi112.stderr:269331: hinfo_key 2024-09-13T14:56:39.739 INFO:tasks.workunit.client.0.smithi112.stderr:269331: snapset 2024-09-13T14:56:39.739 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-13T14:56:39.739 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:56:39.739 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:56:39.739 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-13T14:56:39.740 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:56:39.740 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-13T14:56:39.740 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:56:39.740 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:56:39.742 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:56:39.742 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:56:39.742 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-13T14:56:39.742 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-13T14:56:39.743 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:56:39.743 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:56:39.743 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:56:39.743 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:56:39.743 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:56:39.743 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:56:39.743 INFO:tasks.workunit.client.0.smithi112.stderr:269331: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:56:39.743 INFO:tasks.workunit.client.0.smithi112.stderr:269331: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:56:39.743 INFO:tasks.workunit.client.0.smithi112.stderr:269331: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:56:39.743 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:56:39.743 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:56:39.743 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:56:39.743 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:56:39.744 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:56:39.744 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:56:39.744 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:56:39.744 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:56:39.744 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:56:39.744 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:56:39.744 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:56:39.744 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-13T14:56:39.744 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-13T14:56:39.744 INFO:tasks.workunit.client.0.smithi112.stderr:269331: start osd.5 2024-09-13T14:56:39.795 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:56:39.796 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:56:39.796 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T14:56:39.796 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:56:39.796 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:56:39.796 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:56:39.796 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:56:39.796 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T14:56:39.796 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:56:39.796 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T14:56:39.796 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:56:39.796 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:56:39.796 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:56:39.797 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:56:39.797 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:56:39.797 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:56:39.797 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-13T14:56:39.797 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:56:39.797 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:56:39.797 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T14:56:39.797 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:56:39.797 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:56:39.797 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-13T14:56:39.797 INFO:tasks.workunit.client.0.smithi112.stderr:269329: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-13T14:56:39.797 INFO:tasks.workunit.client.0.smithi112.stderr:269329: _ 2024-09-13T14:56:39.797 INFO:tasks.workunit.client.0.smithi112.stderr:269329: hinfo_key 2024-09-13T14:56:39.798 INFO:tasks.workunit.client.0.smithi112.stderr:269329: snapset 2024-09-13T14:56:39.798 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T14:56:39.798 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:56:39.798 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:56:39.798 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T14:56:39.798 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:56:39.798 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T14:56:39.798 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:56:39.798 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:56:39.799 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:56:39.799 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:56:39.799 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T14:56:39.799 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T14:56:39.799 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:56:39.799 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:56:39.799 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:56:39.799 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:56:39.799 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:56:39.799 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:56:39.799 INFO:tasks.workunit.client.0.smithi112.stderr:269329: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:56:39.799 INFO:tasks.workunit.client.0.smithi112.stderr:269329: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:56:39.799 INFO:tasks.workunit.client.0.smithi112.stderr:269329: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:56:39.799 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:56:39.800 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:56:39.800 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:56:39.800 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:56:39.800 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:56:39.800 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:56:39.800 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:56:39.800 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:56:39.800 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:56:39.800 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:56:39.800 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:56:39.800 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T14:56:39.800 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T14:56:39.800 INFO:tasks.workunit.client.0.smithi112.stderr:269329: start osd.3 2024-09-13T14:56:44.718 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:56:44.719 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T14:56:44.719 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T14:56:44.719 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:56:44.719 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:56:44.719 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:56:44.719 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 2024-09-13T14:56:39.825+0000 7fc15b62b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:44.719 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 2024-09-13T14:56:39.829+0000 7fc15b62b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:44.719 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 2024-09-13T14:56:39.829+0000 7fc15b62b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:44.719 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T14:56:44.719 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:56:44.719 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T14:56:44.719 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:56:44.719 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:56:44.719 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:44.720 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:56:44.720 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 0 2024-09-13T14:56:44.720 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:44.720 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:56:44.720 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:44.720 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 2024-09-13T14:56:41.197+0000 7fc15b62b7c0 -1 Falling back to public interface 2024-09-13T14:56:44.720 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:44.720 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:44.720 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:56:44.720 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 1 2024-09-13T14:56:44.720 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:44.720 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:56:44.720 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:44.720 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:44.720 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:44.721 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:56:44.721 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 2 2024-09-13T14:56:44.721 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:44.721 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:56:44.721 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:44.721 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:44.721 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:44.721 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:56:44.721 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 3 2024-09-13T14:56:44.721 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:44.721 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:56:44.952 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.very_ops 2024-09-13T14:56:44.952 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-13T14:56:44.953 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-13T14:56:44.953 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:56:44.953 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:56:44.953 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:56:44.953 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 2024-09-13T14:56:39.777+0000 7f0d3d1bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:44.953 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 2024-09-13T14:56:39.789+0000 7f0d3d1bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:44.953 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 2024-09-13T14:56:39.793+0000 7f0d3d1bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:44.953 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-13T14:56:44.953 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:56:44.953 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-13T14:56:44.953 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:56:44.953 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:56:44.953 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:44.953 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:56:44.954 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 0 2024-09-13T14:56:44.954 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:44.954 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:44.954 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:44.954 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 2024-09-13T14:56:40.841+0000 7f0d3d1bf7c0 -1 Falling back to public interface 2024-09-13T14:56:44.954 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:44.954 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:44.954 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:56:44.954 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 1 2024-09-13T14:56:44.954 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:44.954 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:44.954 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:44.954 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:44.954 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:44.954 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:56:44.955 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 2 2024-09-13T14:56:44.955 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:44.955 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:44.955 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:44.955 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:44.955 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:44.955 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:56:44.955 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 3 2024-09-13T14:56:44.955 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:44.955 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:44.955 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:47.869 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 2024-09-13T14:56:44.949+0000 7f0d3d1bf7c0 -1 osd.5 117 log_to_monitors true 2024-09-13T14:56:47.869 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:47.870 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:47.870 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:56:47.870 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 4 2024-09-13T14:56:47.870 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:47.870 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:47.870 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:47.870 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:47.870 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:47.870 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:56:47.870 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 5 2024-09-13T14:56:47.870 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:47.871 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-13T14:56:47.871 INFO:tasks.workunit.client.0.smithi112.stderr:269331: osd.5 up in weight 1 up_from 122 up_thru 122 down_at 118 last_clean_interval [115,117) [v2:127.0.0.1:6826/947092183,v1:127.0.0.1:6827/947092183] [v2:127.0.0.1:6828/947092183,v1:127.0.0.1:6829/947092183] exists,up 3df67c99-00a1-4d12-b468-6f9b9faac1b1 2024-09-13T14:56:47.871 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:56:47.871 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:56:47.871 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:56:47.871 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:56:47.871 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:56:47.871 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:56:47.871 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:56:47.871 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:56:47.872 INFO:tasks.workunit.client.0.smithi112.stderr:269331: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:56:47.872 INFO:tasks.workunit.client.0.smithi112.stderr:269331: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:56:47.872 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:56:47.872 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:56:47.872 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:56:47.872 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:56:47.872 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:56:47.872 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:56:47.872 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:56:47.872 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:56:47.872 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:56:47.872 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:56:47.873 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 1 2024-09-13T14:56:47.873 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 2 2024-09-13T14:56:47.873 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 3 2024-09-13T14:56:47.873 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 4 2024-09-13T14:56:47.873 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 5 2024-09-13T14:56:47.873 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 6 2024-09-13T14:56:47.873 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 7 2024-09-13T14:56:47.873 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 8 2024-09-13T14:56:47.873 INFO:tasks.workunit.client.0.smithi112.stderr:269331: 9' 2024-09-13T14:56:47.873 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:56:48.595 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:48.595 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 2024-09-13T14:56:45.249+0000 7fc15b62b7c0 -1 osd.3 117 log_to_monitors true 2024-09-13T14:56:48.595 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:48.595 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:48.595 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:56:48.595 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 4 2024-09-13T14:56:48.595 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:48.596 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:56:48.596 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:56:48.596 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:56:48.596 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:56:48.596 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:56:48.596 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 5 2024-09-13T14:56:48.596 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:56:48.596 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T14:56:48.596 INFO:tasks.workunit.client.0.smithi112.stderr:269329: osd.3 up in weight 1 up_from 122 up_thru 115 down_at 118 last_clean_interval [113,117) [v2:127.0.0.1:6874/2232668970,v1:127.0.0.1:6875/2232668970] [v2:127.0.0.1:6876/2232668970,v1:127.0.0.1:6877/2232668970] exists,up 858e19ad-bbe8-4eeb-8da5-f6f455b0cb32 2024-09-13T14:56:48.596 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:56:48.596 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:56:48.596 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:56:48.596 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:56:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:56:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:56:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:56:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:56:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:269329: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:56:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:269329: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:56:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:56:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:56:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:56:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:56:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:56:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:56:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:56:48.598 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:56:48.598 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:56:48.598 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:56:48.598 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 1 2024-09-13T14:56:48.598 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 2 2024-09-13T14:56:48.598 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 3 2024-09-13T14:56:48.598 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 4 2024-09-13T14:56:48.598 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 5 2024-09-13T14:56:48.598 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 6 2024-09-13T14:56:48.598 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 7 2024-09-13T14:56:48.598 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 8 2024-09-13T14:56:48.598 INFO:tasks.workunit.client.0.smithi112.stderr:269329: 9' 2024-09-13T14:56:49.592 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalon/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:56:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836597 2024-09-13T14:56:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836597 2024-09-13T14:56:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597' 2024-09-13T14:56:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:56:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673070 2024-09-13T14:56:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673070 2024-09-13T14:56:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673070' 2024-09-13T14:56:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:56:49.593 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509542 2024-09-13T14:56:49.594 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509542 2024-09-13T14:56:49.594 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509542' 2024-09-13T14:56:49.594 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:49.594 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:56:49.594 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=523986010114 2024-09-13T14:56:49.594 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 523986010114 2024-09-13T14:56:49.594 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509542 3-523986010114' 2024-09-13T14:56:49.594 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:49.594 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:56:49.594 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182490 2024-09-13T14:56:49.594 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182490 2024-09-13T14:56:49.594 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509542 3-523986010114 4-107374182490' 2024-09-13T14:56:49.594 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:49.595 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:56:49.595 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=523986010114 2024-09-13T14:56:49.595 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 523986010114 2024-09-13T14:56:49.595 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509542 3-523986010114 4-107374182490 5-523986010114' 2024-09-13T14:56:49.595 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:49.595 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:56:49.875 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=1503238554e/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:56:49.875 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:49.875 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:56:49.875 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836598 2024-09-13T14:56:49.875 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836598 2024-09-13T14:56:49.875 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598' 2024-09-13T14:56:49.875 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:49.876 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:56:49.876 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673071 2024-09-13T14:56:49.876 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673071 2024-09-13T14:56:49.876 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673071' 2024-09-13T14:56:49.876 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:49.876 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T14:56:49.876 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509543 2024-09-13T14:56:49.876 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509543 2024-09-13T14:56:49.876 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673071 2-64424509543' 2024-09-13T14:56:49.876 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:49.876 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T14:56:49.876 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=523986010115 2024-09-13T14:56:49.877 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 523986010115 2024-09-13T14:56:49.877 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673071 2-64424509543 3-523986010115' 2024-09-13T14:56:49.877 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:49.877 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-13T14:56:49.877 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182491 2024-09-13T14:56:49.877 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182491 2024-09-13T14:56:49.877 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673071 2-64424509543 3-523986010115 4-107374182491' 2024-09-13T14:56:49.877 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:49.877 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-13T14:56:49.877 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=523986010115 2024-09-13T14:56:49.877 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 523986010115 2024-09-13T14:56:49.877 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673071 2-64424509543 3-523986010115 4-107374182491 5-523986010115' 2024-09-13T14:56:49.877 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:49.878 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-13T14:56:50.352 INFO:tasks.workunit.client.0.smithi112.stderr:269329:35 2024-09-13T14:56:50.352 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855435 2024-09-13T14:56:50.352 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509542 3-523986010114 4-107374182490 5-523986010114 6-150323855435' 2024-09-13T14:56:50.353 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:50.353 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:56:50.353 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691907 2024-09-13T14:56:50.353 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691907 2024-09-13T14:56:50.353 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509542 3-523986010114 4-107374182490 5-523986010114 6-150323855435 7-171798691907' 2024-09-13T14:56:50.353 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:50.353 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:56:50.353 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528384 2024-09-13T14:56:50.353 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528384 2024-09-13T14:56:50.353 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509542 3-523986010114 4-107374182490 5-523986010114 6-150323855435 7-171798691907 8-193273528384' 2024-09-13T14:56:50.353 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:50.353 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:56:50.353 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566095 2024-09-13T14:56:50.354 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566095 2024-09-13T14:56:50.354 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673070 2-64424509542 3-523986010114 4-107374182490 5-523986010114 6-150323855435 7-171798691907 8-193273528384 9-450971566095' 2024-09-13T14:56:50.354 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:50.354 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836597 2024-09-13T14:56:50.354 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:50.354 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:56:50.354 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836597 2024-09-13T14:56:50.354 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:50.354 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836597 2024-09-13T14:56:50.354 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836597' 2024-09-13T14:56:50.354 INFO:tasks.workunit.client.0.smithi112.stderr:269331: waiting osd.0 seq 21474836597 2024-09-13T14:56:50.354 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:56:50.354 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836598 -lt 21474836597 2024-09-13T14:56:50.354 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:50.355 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673070 2024-09-13T14:56:50.355 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:50.639 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/stand /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855436 2024-09-13T14:56:50.639 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855436 2024-09-13T14:56:50.639 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673071 2-64424509543 3-523986010115 4-107374182491 5-523986010115 6-150323855436' 2024-09-13T14:56:50.639 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:50.640 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-13T14:56:50.640 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691908 2024-09-13T14:56:50.640 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691908 2024-09-13T14:56:50.640 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673071 2-64424509543 3-523986010115 4-107374182491 5-523986010115 6-150323855436 7-171798691908' 2024-09-13T14:56:50.640 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:50.640 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-13T14:56:50.640 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528385 2024-09-13T14:56:50.640 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528385 2024-09-13T14:56:50.640 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673071 2-64424509543 3-523986010115 4-107374182491 5-523986010115 6-150323855436 7-171798691908 8-193273528385' 2024-09-13T14:56:50.640 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:56:50.640 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-13T14:56:50.640 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566096 2024-09-13T14:56:50.640 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566096 2024-09-13T14:56:50.641 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673071 2-64424509543 3-523986010115 4-107374182491 5-523986010115 6-150323855436 7-171798691908 8-193273528385 9-450971566096' 2024-09-13T14:56:50.641 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:50.641 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836598 2024-09-13T14:56:50.641 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:50.641 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:56:50.641 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836598 2024-09-13T14:56:50.641 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:50.641 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836598 2024-09-13T14:56:50.641 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836598' 2024-09-13T14:56:50.641 INFO:tasks.workunit.client.0.smithi112.stderr:269329: waiting osd.0 seq 21474836598 2024-09-13T14:56:50.641 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:56:50.641 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836598 -lt 21474836598 2024-09-13T14:56:50.641 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:50.641 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673071 2024-09-13T14:56:51.432 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/calone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:56:51.433 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673070 2024-09-13T14:56:51.433 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:51.433 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673070 2024-09-13T14:56:51.433 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673070' 2024-09-13T14:56:51.433 INFO:tasks.workunit.client.0.smithi112.stderr:269331: waiting osd.1 seq 42949673070 2024-09-13T14:56:51.433 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:56:51.433 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673071 -lt 42949673070 2024-09-13T14:56:51.433 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:51.433 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509542 2024-09-13T14:56:51.433 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:51.433 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:56:51.433 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509542 2024-09-13T14:56:51.433 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:51.434 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509542 2024-09-13T14:56:51.434 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509542' 2024-09-13T14:56:51.434 INFO:tasks.workunit.client.0.smithi112.stderr:269331: waiting osd.2 seq 64424509542 2024-09-13T14:56:51.434 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:56:51.434 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509543 -lt 64424509542 2024-09-13T14:56:51.434 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:51.434 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-523986010114 2024-09-13T14:56:51.434 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:51.434 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:56:51.434 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-523986010114 2024-09-13T14:56:51.434 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:51.434 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=523986010114 2024-09-13T14:56:51.435 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 523986010114' 2024-09-13T14:56:51.435 INFO:tasks.workunit.client.0.smithi112.stderr:269331: waiting osd.3 seq 523986010114 2024-09-13T14:56:51.435 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:56:51.435 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 523986010115 -lt 523986010114 2024-09-13T14:56:51.435 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:51.435 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:51.435 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182490 2024-09-13T14:56:51.436 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:56:51.436 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182490 2024-09-13T14:56:51.436 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:51.436 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182490 2024-09-13T14:56:51.712 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephteeph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:51.712 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:56:51.712 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673071 2024-09-13T14:56:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673071 2024-09-13T14:56:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673071' 2024-09-13T14:56:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:269329: waiting osd.1 seq 42949673071 2024-09-13T14:56:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:56:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673071 -lt 42949673071 2024-09-13T14:56:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509543 2024-09-13T14:56:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T14:56:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509543 2024-09-13T14:56:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509543 2024-09-13T14:56:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509543' 2024-09-13T14:56:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:269329: waiting osd.2 seq 64424509543 2024-09-13T14:56:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T14:56:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509543 -lt 64424509543 2024-09-13T14:56:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-523986010115 2024-09-13T14:56:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T14:56:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-523986010115 2024-09-13T14:56:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=523986010115 2024-09-13T14:56:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 523986010115' 2024-09-13T14:56:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:269329: waiting osd.3 seq 523986010115 2024-09-13T14:56:51.715 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T14:56:51.715 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 523986010115 -lt 523986010115 2024-09-13T14:56:51.715 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:51.715 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182491 2024-09-13T14:56:51.715 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:51.715 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-13T14:56:51.715 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182491 2024-09-13T14:56:51.715 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:52.721 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.cst/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182490' 2024-09-13T14:56:52.721 INFO:tasks.workunit.client.0.smithi112.stderr:269331: waiting osd.4 seq 107374182490 2024-09-13T14:56:52.721 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:56:52.722 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182491 -lt 107374182490 2024-09-13T14:56:52.722 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:52.722 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-523986010114 2024-09-13T14:56:52.722 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:52.722 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:56:52.722 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-523986010114 2024-09-13T14:56:52.722 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:52.722 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=523986010114 2024-09-13T14:56:52.722 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 523986010114' 2024-09-13T14:56:52.722 INFO:tasks.workunit.client.0.smithi112.stderr:269331: waiting osd.5 seq 523986010114 2024-09-13T14:56:52.722 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:56:52.722 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 523986010115 -lt 523986010114 2024-09-13T14:56:52.723 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:52.723 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855435 2024-09-13T14:56:52.723 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:52.723 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:56:52.723 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855435 2024-09-13T14:56:52.723 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:52.723 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855435 2024-09-13T14:56:52.723 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855435' 2024-09-13T14:56:52.723 INFO:tasks.workunit.client.0.smithi112.stderr:269331: waiting osd.6 seq 150323855435 2024-09-13T14:56:52.723 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:56:52.723 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855436 -lt 150323855435 2024-09-13T14:56:52.723 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:52.723 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691907 2024-09-13T14:56:52.724 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:52.724 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:56:52.724 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691907 2024-09-13T14:56:52.724 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:52.724 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691907 2024-09-13T14:56:52.724 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691907' 2024-09-13T14:56:52.724 INFO:tasks.workunit.client.0.smithi112.stderr:269331: waiting osd.7 seq 171798691907 2024-09-13T14:56:52.724 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:56:52.724 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691908 -lt 171798691907 2024-09-13T14:56:53.076 INFO:tasks.workunit.client.0.smithi112.stderr:269331:lient.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182491 2024-09-13T14:56:53.077 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182491' 2024-09-13T14:56:53.077 INFO:tasks.workunit.client.0.smithi112.stderr:269329: waiting osd.4 seq 107374182491 2024-09-13T14:56:53.077 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-13T14:56:53.077 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182491 -lt 107374182491 2024-09-13T14:56:53.077 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:53.077 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-523986010115 2024-09-13T14:56:53.077 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:53.077 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-13T14:56:53.077 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-523986010115 2024-09-13T14:56:53.077 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:53.077 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=523986010115 2024-09-13T14:56:53.077 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 523986010115' 2024-09-13T14:56:53.078 INFO:tasks.workunit.client.0.smithi112.stderr:269329: waiting osd.5 seq 523986010115 2024-09-13T14:56:53.078 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-13T14:56:53.078 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 523986010116 -lt 523986010115 2024-09-13T14:56:53.078 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:53.078 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855436 2024-09-13T14:56:53.078 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:53.078 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-13T14:56:53.078 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855436 2024-09-13T14:56:53.078 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:53.078 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855436 2024-09-13T14:56:53.078 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855436' 2024-09-13T14:56:53.078 INFO:tasks.workunit.client.0.smithi112.stderr:269329: waiting osd.6 seq 150323855436 2024-09-13T14:56:53.078 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-13T14:56:53.078 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855436 -lt 150323855436 2024-09-13T14:56:53.079 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:53.079 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691908 2024-09-13T14:56:53.079 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:53.079 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-13T14:56:53.079 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691908 2024-09-13T14:56:53.079 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:53.079 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691908 2024-09-13T14:56:53.079 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691908' 2024-09-13T14:56:53.079 INFO:tasks.workunit.client.0.smithi112.stderr:269329: waiting osd.7 seq 171798691908 2024-09-13T14:56:53.079 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-13T14:56:54.113 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:54.113 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528384 2024-09-13T14:56:54.113 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:54.113 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:56:54.113 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528384 2024-09-13T14:56:54.114 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:54.114 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528384 2024-09-13T14:56:54.114 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528384' 2024-09-13T14:56:54.114 INFO:tasks.workunit.client.0.smithi112.stderr:269331: waiting osd.8 seq 193273528384 2024-09-13T14:56:54.114 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:56:54.114 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528385 -lt 193273528384 2024-09-13T14:56:54.114 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:54.114 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566095 2024-09-13T14:56:54.114 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:54.114 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:56:54.115 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566095 2024-09-13T14:56:54.115 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:54.115 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566095 2024-09-13T14:56:54.115 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566095' 2024-09-13T14:56:54.115 INFO:tasks.workunit.client.0.smithi112.stderr:269331: waiting osd.9 seq 450971566095 2024-09-13T14:56:54.115 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:56:54.115 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566097 -lt 450971566095 2024-09-13T14:56:54.115 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:56:54.115 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:56:54.115 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:56:54.115 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:56:54.115 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:56:54.116 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:56:54.116 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:56:54.116 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:56:54.116 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:56:54.116 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:56:54.116 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:56:54.116 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:56:54.527 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:56:54.527 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:56:54.527 INFO:tasks.workunit.client.0.smithi112.stderr:269331: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:56:54.527 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:56:54.527 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:56:54.527 INFO:tasks.workunit.client.0.smithi112.stderr:269331: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:56:54.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:56:54.554 INFO:tasks.workunit.client.0.smithi112.stderr:.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691908 -lt 171798691908 2024-09-13T14:56:54.554 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:54.554 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528385 2024-09-13T14:56:54.554 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:54.554 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-13T14:56:54.554 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528385 2024-09-13T14:56:54.554 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:54.554 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528385 2024-09-13T14:56:54.554 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528385' 2024-09-13T14:56:54.554 INFO:tasks.workunit.client.0.smithi112.stderr:269329: waiting osd.8 seq 193273528385 2024-09-13T14:56:54.554 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-13T14:56:54.556 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528385 -lt 193273528385 2024-09-13T14:56:54.556 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:56:54.556 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566096 2024-09-13T14:56:54.556 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:56:54.556 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-13T14:56:54.556 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566096 2024-09-13T14:56:54.556 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:56:54.556 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566096 2024-09-13T14:56:54.556 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566096' 2024-09-13T14:56:54.556 INFO:tasks.workunit.client.0.smithi112.stderr:269329: waiting osd.9 seq 450971566096 2024-09-13T14:56:54.556 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-13T14:56:54.556 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566097 -lt 450971566096 2024-09-13T14:56:54.557 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:56:54.557 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:56:54.557 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:56:54.557 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T14:56:54.557 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:56:54.557 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:56:54.557 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:56:54.557 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:56:54.557 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:56:54.557 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:56:54.557 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:56:54.956 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T14:56:54.956 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:56:54.956 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:56:54.956 INFO:tasks.workunit.client.0.smithi112.stderr:269329: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:56:54.956 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T14:56:54.956 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:56:54.956 INFO:tasks.workunit.client.0.smithi112.stderr:269329: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:56:54.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T14:56:54.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T14:56:54.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 269328 2024-09-13T14:56:54.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T14:56:54.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T14:56:54.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T14:56:54.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-13T14:56:54.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-13T14:56:54.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:56:55.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:56:55.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T14:56:55.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:56:55.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:56:55.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:56:55.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:56:55.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:56:55.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:56:55.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:56:55.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:56:55.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:56:55.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:56:55.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:56:55.320 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:56:55.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:56:55.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:56:55.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:56:55.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:56:55.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:56:55.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:56:55.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:56:55.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:56:55.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:56:55.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:56:55.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:56:55.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:56:55.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:56:55.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:56:55.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:56:55.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:56:55.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T14:56:55.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T14:56:55.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T14:56:55.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T14:56:55.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:56:55.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:56:55.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:56:55.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:56:55.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:56:55.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:56:55.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:56:55.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:56:55.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:56:55.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:56:55.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:56:55.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:56:55.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:56:55.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:56:55.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:56:55.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:56:55.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:56:55.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:56:55.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:56:55.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:56:55.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:56:55.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:56:55.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:56:55.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:56:55.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:56:55.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:56:55.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:56:55.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:56:55.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T14:56:55.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:56:55.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:56:55.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:56:55.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T14:56:55.415 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:56:55.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T14:56:55.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:56:55.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T14:56:55.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-09-13T14:56:55.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-09-13T14:56:55.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-09-13T14:56:55.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-13T14:56:55.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T14:56:55.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T14:56:55.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T14:56:55.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T14:56:55.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T14:56:55.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-13T14:56:55.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T14:56:55.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:56:55.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:56:55.876 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:56:55.876 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:56:55.876 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:56:55.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:56:55.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-13T14:56:56.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T14:56:56.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T14:56:56.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:56:56.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:56:56.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T14:56:56.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T14:56:56.071 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:56:56.072 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:56:56.072 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:56:56.073 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:56:56.073 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:56:56.073 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:56:56.073 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:56:56.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:56:56.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T14:56:56.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T14:56:56.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:56:56.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:56:56.180 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T14:56:56.180 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T14:56:56.180 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:56:56.180 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:56:56.181 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:56:56.181 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:56:56.181 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:56:56.181 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:56:56.181 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:56:56.181 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:56:56.181 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T14:56:56.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-09-13T14:56:56.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T14:56:56.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T14:56:56.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T14:56:56.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T14:56:56.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T14:56:56.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T14:56:57.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T14:56:57.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:56:57.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:56:57.135 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:56:57.135 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:56:57.135 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:56:57.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:56:57.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:56:57.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:56:57.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-09-13T14:56:57.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:56:57.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:56:57.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T14:56:57.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:56:57.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:56:57.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:56:57.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:56:57.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:56:57.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:56:57.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:56:57.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:56:57.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:56:57.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:56:57.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:56:57.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:56:57.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:56:57.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:56:57.175 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:56:57.175 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:56:57.175 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:56:57.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:56:57.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:56:57.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:56:57.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:56:57.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:56:57.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:56:57.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:56:57.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:56:57.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:56:57.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:56:57.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:56:57.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:56:57.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:56:57.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:56:57.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:56:57.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1759c657-45cf-4819-aa38-ad757288db3d 2024-09-13T14:56:57.177 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 1759c657-45cf-4819-aa38-ad757288db3d 2024-09-13T14:56:57.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1759c657-45cf-4819-aa38-ad757288db3d' 2024-09-13T14:56:57.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:56:57.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC5UuRmgxSXCxAAT1kbJ0QrTFfSW9OncDYNEA== 2024-09-13T14:56:57.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC5UuRmgxSXCxAAT1kbJ0QrTFfSW9OncDYNEA=="}' 2024-09-13T14:56:57.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1759c657-45cf-4819-aa38-ad757288db3d -i td/osd-scrub-repair/0/new.json 2024-09-13T14:56:57.439 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:56:57.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T14:56:57.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC5UuRmgxSXCxAAT1kbJ0QrTFfSW9OncDYNEA== --osd-uuid 1759c657-45cf-4819-aa38-ad757288db3d 2024-09-13T14:56:57.485 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:56:57.480+0000 7fe7f68927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:57.489 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:56:57.484+0000 7fe7f68927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:57.490 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:56:57.488+0000 7fe7f68927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:56:57.491 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:56:57.488+0000 7fe7f68927c0 -1 bdev(0x556459fedc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:56:57.491 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:56:57.488+0000 7fe7f68927c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T14:57:03.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T14:57:03.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:57:03.403 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:57:03.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T14:57:03.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:57:03.812 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:57:03.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T14:57:03.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:57:03.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:57:03.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:57:03.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:57:03.845 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:57:03.840+0000 7fddcc7167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:57:03.848 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:57:03.844+0000 7fddcc7167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:57:03.849 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:57:03.844+0000 7fddcc7167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:57:04.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T14:57:04.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:57:04.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:57:04.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:57:04.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:57:04.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:04.172 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:57:04.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:57:04.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:04.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:04.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:05.173 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:57:05.168+0000 7fddcc7167c0 -1 Falling back to public interface 2024-09-13T14:57:05.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:05.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:05.482 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:57:05.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:57:05.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:05.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:05.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:06.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:06.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:06.815 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:57:06.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:57:06.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:06.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:07.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:07.844 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:57:07.840+0000 7fddcc7167c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:57:08.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:08.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:08.156 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:57:08.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:57:08.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:08.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:08.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:09.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:09.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:09.543 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:57:09.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:57:09.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:09.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:09.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:10.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:10.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:10.878 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:57:10.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:57:10.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:10.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:11.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:12.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:12.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:12.210 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:57:12.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:57:12.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:12.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:12.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:13.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:13.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:13.556 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:57:13.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:57:13.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:13.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:13.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:14.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:14.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:14.879 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:57:14.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:57:14.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:14.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:15.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:16.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:16.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:16.206 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:57:16.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:57:16.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:16.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:16.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:17.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:17.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:17.563 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:57:17.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:57:17.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:17.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:17.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:18.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:18.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:18.902 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:57:18.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:57:18.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:18.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:19.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:20.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:20.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:20.228 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:57:20.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:57:20.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:20.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:20.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:21.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:21.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:21.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:57:21.934 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:57:21.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:21.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:22.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:23.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:23.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:23.866 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:57:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:57:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:23.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:24.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:25.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:25.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:25.232 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:57:25.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:57:25.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:25.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:26.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:27.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:27.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:27.012 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:57:27.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:57:27.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:27.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:27.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:28.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:28.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:28.721 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:57:28.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:57:28.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:28.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:29.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:30.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:30.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:30.232 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:57:30.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:57:30.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:30.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:57:30.572 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3152189719,v1:127.0.0.1:6803/3152189719] [v2:127.0.0.1:6804/3152189719,v1:127.0.0.1:6805/3152189719] exists,up 1759c657-45cf-4819-aa38-ad757288db3d 2024-09-13T14:57:30.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:57:30.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:57:30.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:57:30.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-09-13T14:57:30.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:57:30.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:57:30.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T14:57:30.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:57:30.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:57:30.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:57:30.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:57:30.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:57:30.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:57:30.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:57:30.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:57:30.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:57:30.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:57:30.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:57:30.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:57:30.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:57:30.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:57:30.580 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:57:30.580 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:57:30.580 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:57:30.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:57:30.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:57:30.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:57:30.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:57:30.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:57:30.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:57:30.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:57:30.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:57:30.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:57:30.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:57:30.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:57:30.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:57:30.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:57:30.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:57:30.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:57:30.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=efa940ef-fa1a-4793-9fef-ca428c78f418 2024-09-13T14:57:30.582 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 efa940ef-fa1a-4793-9fef-ca428c78f418 2024-09-13T14:57:30.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 efa940ef-fa1a-4793-9fef-ca428c78f418' 2024-09-13T14:57:30.583 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:57:30.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDaUuRmcNrUIxAAYCSRGWD99p7kp3VAtHXvtg== 2024-09-13T14:57:30.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDaUuRmcNrUIxAAYCSRGWD99p7kp3VAtHXvtg=="}' 2024-09-13T14:57:30.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new efa940ef-fa1a-4793-9fef-ca428c78f418 -i td/osd-scrub-repair/1/new.json 2024-09-13T14:57:30.984 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:57:31.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T14:57:31.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDaUuRmcNrUIxAAYCSRGWD99p7kp3VAtHXvtg== --osd-uuid efa940ef-fa1a-4793-9fef-ca428c78f418 2024-09-13T14:57:31.041 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:57:31.040+0000 7fb4528ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:57:31.095 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:57:31.096+0000 7fb4528ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:57:31.097 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:57:31.096+0000 7fb4528ec7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:57:31.097 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:57:31.096+0000 7fb4528ec7c0 -1 bdev(0x559410056800 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:57:31.098 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:57:31.096+0000 7fb4528ec7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T14:57:37.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T14:57:37.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:57:37.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T14:57:37.466 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:57:37.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:57:37.873 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:57:37.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T14:57:37.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:57:37.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:57:37.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:57:37.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:57:37.906 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:57:37.903+0000 7f74df4d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:57:37.908 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:57:37.907+0000 7f74df4d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:57:37.909 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:57:37.907+0000 7f74df4d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:57:38.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T14:57:38.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:57:38.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:57:38.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:57:38.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:57:38.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:38.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:57:38.240 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:57:38.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:38.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:57:38.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:39.181 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:57:39.179+0000 7f74df4d77c0 -1 Falling back to public interface 2024-09-13T14:57:39.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:39.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:39.535 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:57:39.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:57:39.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:39.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:57:39.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:40.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:40.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:40.867 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:57:40.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:57:40.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:40.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:57:41.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:42.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:42.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:42.184 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:57:42.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:57:42.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:42.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:57:42.190 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:57:42.191+0000 7f74df4d77c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:57:42.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:43.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:43.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:43.732 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:57:43.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:57:43.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:43.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:57:44.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:45.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:45.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:45.073 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:57:45.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:57:45.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:45.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:57:45.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:46.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:46.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:46.413 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:57:46.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:57:46.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:46.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:57:46.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:47.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:47.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:47.751 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:57:47.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:57:47.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:47.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:57:48.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:49.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:49.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:49.109 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:57:49.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:57:49.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:49.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:57:49.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:50.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:50.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:50.449 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:57:50.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:57:50.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:50.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:57:50.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:51.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:51.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:51.773 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:57:51.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:57:51.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:51.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:57:52.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:53.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:53.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:53.135 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:57:53.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:57:53.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:53.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:57:53.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:54.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:54.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:54.472 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:57:54.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:57:54.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:54.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:57:55.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:56.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:56.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:56.159 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:57:56.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:57:56.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:56.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:57:56.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:57.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:57.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:57.992 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:57:57.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:57:57.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:57.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:57:58.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:57:59.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:57:59.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:57:59.449 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:57:59.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:57:59.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:57:59.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:58:00.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:58:01.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:58:01.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:01.292 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:58:01.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:58:01.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:01.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:58:01.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:58:02.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:58:02.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:02.740 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:58:02.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:58:02.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:02.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:58:03.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:58:04.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:58:04.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:04.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T14:58:04.490 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T14:58:04.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:04.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:58:04.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:58:05.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:58:05.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:05.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T14:58:05.904 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T14:58:05.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:05.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:58:06.242 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2044096373,v1:127.0.0.1:6811/2044096373] [v2:127.0.0.1:6812/2044096373,v1:127.0.0.1:6813/2044096373] exists,up efa940ef-fa1a-4793-9fef-ca428c78f418 2024-09-13T14:58:06.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:58:06.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:58:06.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:58:06.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-09-13T14:58:06.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T14:58:06.550 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' does not exist 2024-09-13T14:58:06.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T14:58:06.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T14:58:06.959 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T14:58:06.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T14:58:07.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T14:58:09.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-09-13T14:58:09.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:58:09.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:58:09.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:58:09.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:58:09.723 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:58:09.723 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:58:09.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:58:09.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:58:09.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:58:09.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:58:09.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:58:09.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:58:09.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:58:09.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:58:09.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:58:10.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:58:10.213 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T14:58:10.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:58:10.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:58:10.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:58:10.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836493 2024-09-13T14:58:10.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836493 2024-09-13T14:58:10.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836493' 2024-09-13T14:58:10.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:58:10.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:58:10.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-13T14:58:10.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-13T14:58:10.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836493 1-42949672965' 2024-09-13T14:58:10.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:58:10.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836493 2024-09-13T14:58:10.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:58:10.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:58:10.521 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836493 2024-09-13T14:58:10.521 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:58:10.522 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836493 2024-09-13T14:58:10.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836493 2024-09-13T14:58:10.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-09-13T14:58:10.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:58:10.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836493 2024-09-13T14:58:10.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:58:11.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:58:11.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:58:12.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836493 2024-09-13T14:58:12.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:58:12.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-13T14:58:12.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:58:12.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:58:12.191 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:58:12.191 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-13T14:58:12.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-13T14:58:12.191 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672965 2024-09-13T14:58:12.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-13T14:58:12.192 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:58:12.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672966 -lt 42949672965 2024-09-13T14:58:12.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:58:12.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:58:12.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:58:12.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T14:58:12.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:58:12.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:58:12.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:58:12.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:58:12.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:58:12.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:58:12.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:58:13.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:58:13.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:58:13.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:58:13.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:58:13.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T14:58:13.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:58:13.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:58:13.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-09-13T14:58:13.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T14:58:13.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-09-13T14:58:13.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-13T14:58:13.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T14:58:13.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T14:58:13.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T14:58:14.019 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T14:58:14.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T14:58:14.471 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T14:58:14.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T14:58:14.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T14:58:14.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-13T14:58:14.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-09-13T14:58:14.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2024-09-13T14:58:14.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T14:58:14.737 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2024-09-13T14:58:14.737 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-13T14:58:14.737 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:58:14.737 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-13T14:58:14.737 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:58:15.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T14:58:15.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-09-13T14:58:15.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T14:58:15.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-09-13T14:58:15.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-13T14:58:15.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-09-13T14:58:15.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-09-13T14:58:15.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T14:58:15.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:58:15.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:58:15.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T14:58:15.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:58:15.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T14:58:15.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:58:15.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:58:15.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T14:58:15.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:58:15.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T14:58:15.426 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:58:15.426 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:58:15.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:58:15.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:58:15.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:58:15.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:58:15.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T14:58:15.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:58:15.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:58:15.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T14:58:15.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:58:15.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:58:15.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-13T14:58:17.684 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-13T14:58:18.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T14:58:18.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:58:18.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:58:18.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T14:58:18.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:58:18.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:58:18.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:58:18.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:58:18.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:58:18.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:58:18.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:58:18.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:58:18.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:58:18.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:58:18.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:58:18.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:58:18.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:58:18.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:58:18.232 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:58:18.232 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:58:18.232 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:58:18.233 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:58:18.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:58:18.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:58:18.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:58:18.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:58:18.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:58:18.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:58:18.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:58:18.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:58:18.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:58:18.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:58:18.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:58:18.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T14:58:18.234 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T14:58:18.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:58:18.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T14:58:18.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T14:58:18.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:58:18.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:58:18.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:58:18.274 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:58:18.274+0000 7fe987a877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:58:18.277 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:58:18.274+0000 7fe987a877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:58:18.278 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:58:18.278+0000 7fe987a877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:58:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T14:58:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:58:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:58:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:58:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:58:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:58:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:58:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:58:18.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:58:19.301 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:58:19.298+0000 7fe987a877c0 -1 Falling back to public interface 2024-09-13T14:58:19.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:58:19.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:19.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:58:19.915 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:58:19.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:19.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:58:20.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:58:21.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:58:21.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:21.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:58:21.241 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:58:21.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:21.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:58:21.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:58:22.253 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:58:22.250+0000 7fe987a877c0 -1 osd.0 21 log_to_monitors true 2024-09-13T14:58:22.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:58:22.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:22.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:58:22.583 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:58:22.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:22.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:58:23.240 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/2873685476,v1:127.0.0.1:6803/2873685476] [v2:127.0.0.1:6804/2873685476,v1:127.0.0.1:6805/2873685476] exists,up 1759c657-45cf-4819-aa38-ad757288db3d 2024-09-13T14:58:23.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:58:23.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:58:23.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:58:23.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:58:23.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:58:23.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:58:23.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:58:23.241 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:58:23.242 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:58:23.242 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:58:23.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:58:23.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:58:23.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:58:23.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:58:23.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:58:23.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:58:23.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:58:23.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:58:23.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:58:23.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:58:23.725 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T14:58:23.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:58:23.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:58:23.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:58:23.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149698 2024-09-13T14:58:23.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149698 2024-09-13T14:58:23.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698' 2024-09-13T14:58:23.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:58:23.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:58:24.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672969 2024-09-13T14:58:24.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672969 2024-09-13T14:58:24.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698 1-42949672969' 2024-09-13T14:58:24.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:58:24.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:58:24.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149698 2024-09-13T14:58:24.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:58:24.117 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149698 2024-09-13T14:58:24.117 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:58:24.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149698 2024-09-13T14:58:24.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-09-13T14:58:24.118 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 111669149698 2024-09-13T14:58:24.119 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:58:24.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 111669149698 2024-09-13T14:58:24.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:58:25.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:58:25.430 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:58:25.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 111669149698 2024-09-13T14:58:25.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:58:26.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T14:58:26.747 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:58:27.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149698 -lt 111669149698 2024-09-13T14:58:27.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:58:27.082 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672969 2024-09-13T14:58:27.082 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:58:27.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:58:27.084 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672969 2024-09-13T14:58:27.084 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:58:27.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672969 2024-09-13T14:58:27.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-13T14:58:27.086 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672969 2024-09-13T14:58:27.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:58:27.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672970 -lt 42949672969 2024-09-13T14:58:27.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:58:27.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:58:27.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:58:27.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T14:58:27.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:58:27.828 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:58:27.829 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:58:27.829 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:58:27.829 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:58:27.829 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:58:27.829 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:58:28.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:58:28.174 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:58:28.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:58:28.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:58:28.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T14:58:28.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:58:28.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:58:28.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-09-13T14:58:28.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-13T14:58:28.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:58:28.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-13T14:58:28.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:58:28.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-09-13T14:58:28.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-09-13T14:58:28.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.3 2024-09-13T14:58:28.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.3 2024-09-13T14:58:28.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T14:58:28.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:58:28.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:58:28.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T14:58:29.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:58:06.926119+0000 2024-09-13T14:58:29.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.3 2024-09-13T14:58:29.543 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.3 on osd.1 to repair 2024-09-13T14:58:29.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.3 2024-09-13T14:58:06.926119+0000 2024-09-13T14:58:29.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.3 2024-09-13T14:58:29.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:58:06.926119+0000 2024-09-13T14:58:29.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:58:29.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:58:29.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:58:29.561 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-13T14:58:29.561 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T14:58:29.561 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:58:29.561 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:58:29.561 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T14:58:29.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:58:06.926119+0000 '>' 2024-09-13T14:58:06.926119+0000 2024-09-13T14:58:29.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:58:30.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:58:30.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:58:30.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-13T14:58:30.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T14:58:30.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:58:30.883 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T14:58:30.883 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:58:31.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:58:06.926119+0000 '>' 2024-09-13T14:58:06.926119+0000 2024-09-13T14:58:31.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:58:32.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:58:32.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:58:32.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-13T14:58:32.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T14:58:32.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:58:32.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:58:32.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T14:58:32.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:58:30.085154+0000 '>' 2024-09-13T14:58:06.926119+0000 2024-09-13T14:58:32.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:58:32.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:58:34.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:58:34.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:58:34.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:58:34.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:58:34.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:58:34.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:58:34.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:58:34.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:58:34.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:58:34.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:58:34.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:58:34.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:58:34.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:58:34.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:58:34.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:58:34.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:58:34.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:58:34.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:58:34.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:58:34.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:58:34.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:58:34.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:58:34.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:58:34.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-13T14:58:35.367 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-13T14:58:35.367 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:58:35.367 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:58:35.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T14:58:35.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:58:35.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:58:35.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:58:35.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:58:35.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:58:35.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:58:35.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:58:35.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:58:35.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:58:35.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:58:35.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:58:35.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:58:35.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:58:35.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:58:35.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:58:35.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:58:35.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:58:35.660 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:58:35.661 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:58:35.661 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:58:35.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:58:35.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:58:35.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:58:35.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:58:35.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:58:35.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:58:35.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:58:35.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:58:35.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:58:35.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:58:35.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:58:35.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:58:35.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:58:35.663 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T14:58:35.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:58:35.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:58:35.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:58:35.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:58:35.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:58:35.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:58:35.693 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:58:35.690+0000 7f9f577957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:58:35.696 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:58:35.694+0000 7f9f577957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:58:35.697 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:58:35.694+0000 7f9f577957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:58:35.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:58:35.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:58:35.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:58:35.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:58:35.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:58:35.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:35.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:58:35.993 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:58:35.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:35.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:58:36.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:58:36.997 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:58:36.994+0000 7f9f577957c0 -1 Falling back to public interface 2024-09-13T14:58:37.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:58:37.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:37.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:58:37.325 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:58:37.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:37.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:58:37.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:58:38.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:58:38.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:38.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:58:38.661 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:58:38.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:38.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:58:38.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:58:39.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:58:39.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:39.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:58:39.997 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:58:39.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:39.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:58:40.211 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:58:40.210+0000 7f9f577957c0 -1 osd.1 28 log_to_monitors true 2024-09-13T14:58:40.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:58:41.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:58:41.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:41.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:58:41.395 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T14:58:41.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:41.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:58:41.758 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/2126094659,v1:127.0.0.1:6811/2126094659] [v2:127.0.0.1:6812/2126094659,v1:127.0.0.1:6813/2126094659] exists,up efa940ef-fa1a-4793-9fef-ca428c78f418 2024-09-13T14:58:41.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:58:41.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:58:41.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:58:41.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:58:41.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:58:41.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:58:41.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:58:41.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:58:41.759 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:58:41.759 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:58:41.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:58:41.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:58:41.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:58:41.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:58:41.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:58:41.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:58:41.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:58:41.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:58:41.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:58:42.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:58:42.200 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T14:58:42.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:58:42.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:58:42.201 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:58:42.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149702 2024-09-13T14:58:42.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149702 2024-09-13T14:58:42.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149702' 2024-09-13T14:58:42.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:58:42.373 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:58:42.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953474 2024-09-13T14:58:42.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953474 2024-09-13T14:58:42.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-09-13T14:58:42.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:58:42.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149702 2024-09-13T14:58:42.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:58:42.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:58:42.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149702 2024-09-13T14:58:42.515 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:58:42.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149702 2024-09-13T14:58:42.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-09-13T14:58:42.516 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 111669149702 2024-09-13T14:58:42.516 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:58:42.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149701 -lt 111669149702 2024-09-13T14:58:42.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:58:43.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:58:43.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:58:44.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149703 -lt 111669149702 2024-09-13T14:58:44.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:58:44.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-137438953474 2024-09-13T14:58:44.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:58:44.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:58:44.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-137438953474 2024-09-13T14:58:44.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:58:44.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953474 2024-09-13T14:58:44.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-09-13T14:58:44.136 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 137438953474 2024-09-13T14:58:44.137 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:58:44.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953474 -lt 137438953474 2024-09-13T14:58:44.464 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:58:44.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:58:44.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:58:44.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T14:58:44.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:58:44.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:58:44.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:58:44.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:58:44.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:58:44.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:58:44.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:58:45.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:58:45.189 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:58:45.190 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:58:45.190 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:58:45.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T14:58:45.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:58:45.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:58:45.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:58:45.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:58:45.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-09-13T14:58:45.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-13T14:58:45.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T14:58:45.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-13T14:58:45.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T14:58:45.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-09-13T14:58:45.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-13T14:58:45.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-09-13T14:58:45.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-09-13T14:58:45.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:58:45.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:58:45.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:58:45.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:58:45.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:58:45.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:58:45.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:58:45.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:58:45.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:58:45.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:58:45.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:58:45.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:58:45.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:58:45.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:58:45.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:58:45.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:58:46.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:58:46.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-13T14:58:46.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:58:46.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:58:46.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:58:46.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:58:46.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:58:46.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-13T14:58:47.771 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-13T14:58:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T14:58:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:58:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:58:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:58:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:58:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:58:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:58:48.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:58:48.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:58:48.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:58:48.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:58:48.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:58:48.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:58:48.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:58:48.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:58:48.306 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:58:48.306 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:58:48.306 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:58:48.308 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:58:48.308 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:58:48.308 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:58:48.308 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:58:48.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:58:48.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:58:48.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:58:48.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:58:48.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:58:48.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:58:48.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:58:48.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:58:48.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:58:48.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:58:48.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:58:48.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:58:48.309 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T14:58:48.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:58:48.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:58:48.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:58:48.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:58:48.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:58:48.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:58:48.344 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:58:48.342+0000 7fd8d6cd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:58:48.347 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:58:48.346+0000 7fd8d6cd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:58:48.348 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:58:48.346+0000 7fd8d6cd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:58:48.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:58:48.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:58:48.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:58:48.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:58:48.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:58:48.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:48.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:58:48.644 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:58:48.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:58:48.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:48.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:58:49.625 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:58:49.622+0000 7fd8d6cd47c0 -1 Falling back to public interface 2024-09-13T14:58:49.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:58:49.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:49.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:58:49.975 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:58:49.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:49.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:58:50.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:58:51.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:58:51.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:51.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:58:51.363 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:58:51.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:51.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:58:51.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:58:52.435 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:58:52.433+0000 7fd8d6cd47c0 -1 osd.1 34 log_to_monitors true 2024-09-13T14:58:52.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:58:52.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:58:52.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:58:52.706 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:58:52.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:58:52.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:58:53.089 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 38 up_thru 32 down_at 35 last_clean_interval [32,34) [v2:127.0.0.1:6810/1065112508,v1:127.0.0.1:6811/1065112508] [v2:127.0.0.1:6812/1065112508,v1:127.0.0.1:6813/1065112508] exists,up efa940ef-fa1a-4793-9fef-ca428c78f418 2024-09-13T14:58:53.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:58:53.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:58:53.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:58:53.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:58:53.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:58:53.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:58:53.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:58:53.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:58:53.091 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:58:53.091 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:58:53.091 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:58:53.091 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:58:53.091 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:58:53.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:58:53.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:58:53.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:58:53.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:58:53.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:58:53.206 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:58:53.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:58:53.578 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T14:58:53.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:58:53.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:58:53.578 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:58:53.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149706 2024-09-13T14:58:53.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149706 2024-09-13T14:58:53.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149706' 2024-09-13T14:58:53.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:58:53.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:58:53.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757250 2024-09-13T14:58:53.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757250 2024-09-13T14:58:53.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149706 1-163208757250' 2024-09-13T14:58:53.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:58:53.857 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:58:53.857 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149706 2024-09-13T14:58:53.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:58:53.858 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149706 2024-09-13T14:58:53.858 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:58:53.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149706 2024-09-13T14:58:53.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149706' 2024-09-13T14:58:53.859 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 111669149706 2024-09-13T14:58:53.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:58:54.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149706 -lt 111669149706 2024-09-13T14:58:54.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:58:54.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-163208757250 2024-09-13T14:58:54.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:58:54.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:58:54.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-163208757250 2024-09-13T14:58:54.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:58:54.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757250 2024-09-13T14:58:54.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2024-09-13T14:58:54.161 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 163208757250 2024-09-13T14:58:54.161 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:58:54.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757250 -lt 163208757250 2024-09-13T14:58:54.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:58:54.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:58:54.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:58:54.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T14:58:54.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:58:54.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:58:54.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:58:54.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:58:54.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:58:54.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:58:54.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:58:55.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:58:55.198 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:58:55.198 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:58:55.199 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:58:55.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T14:58:55.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:58:55.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:58:55.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-09-13T14:58:55.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-13T14:58:55.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T14:58:55.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-13T14:58:55.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T14:58:55.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-09-13T14:58:55.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-09-13T14:58:55.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.3 2024-09-13T14:58:55.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.3 2024-09-13T14:58:55.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T14:58:55.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:58:55.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:58:55.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T14:58:56.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T14:58:30.085154+0000 2024-09-13T14:58:56.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.3 2024-09-13T14:58:56.561 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.3 on osd.1 to repair 2024-09-13T14:58:56.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.3 2024-09-13T14:58:30.085154+0000 2024-09-13T14:58:56.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.3 2024-09-13T14:58:56.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T14:58:30.085154+0000 2024-09-13T14:58:56.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T14:58:56.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T14:58:56.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:58:56.578 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-13T14:58:56.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T14:58:56.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:58:56.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:58:56.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T14:58:56.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:58:30.085154+0000 '>' 2024-09-13T14:58:30.085154+0000 2024-09-13T14:58:56.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T14:58:57.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T14:58:57.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T14:58:57.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-13T14:58:57.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T14:58:57.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T14:58:57.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T14:58:57.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T14:58:58.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T14:58:57.475910+0000 '>' 2024-09-13T14:58:30.085154+0000 2024-09-13T14:58:58.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T14:58:58.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T14:59:00.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:59:00.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T14:59:00.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T14:59:00.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T14:59:00.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T14:59:00.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:59:00.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T14:59:00.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T14:59:00.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T14:59:00.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T14:59:00.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T14:59:00.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:59:00.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:59:00.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:59:00.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:59:00.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:59:00.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:59:00.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-13T14:59:00.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T14:59:00.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T14:59:00.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T14:59:00.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T14:59:00.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:59:00.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-13T14:59:01.070 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-13T14:59:01.071 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T14:59:01.071 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T14:59:01.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T14:59:01.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T14:59:01.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T14:59:01.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T14:59:01.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T14:59:01.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:59:01.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:59:01.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:59:01.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:59:01.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:59:01.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:59:01.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:59:01.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T14:59:01.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T14:59:01.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:59:01.359 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T14:59:01.359 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:59:01.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:59:01.361 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:59:01.361 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:59:01.361 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:59:01.361 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:59:01.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:59:01.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:59:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:59:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:59:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:59:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:59:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:59:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:59:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T14:59:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T14:59:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:59:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T14:59:01.363 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T14:59:01.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:59:01.363 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T14:59:01.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T14:59:01.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T14:59:01.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T14:59:01.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T14:59:01.397 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:01.393+0000 7ff2d14d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:59:01.400 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:01.397+0000 7ff2d14d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:59:01.401 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:01.397+0000 7ff2d14d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:59:01.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T14:59:01.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:59:01.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:59:01.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:59:01.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:59:01.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:01.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:59:01.684 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T14:59:01.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:01.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:59:02.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:02.493 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:02.489+0000 7ff2d14d87c0 -1 Falling back to public interface 2024-09-13T14:59:03.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:03.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:03.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:59:03.049 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T14:59:03.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:03.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:59:03.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:59:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T14:59:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:59:04.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:05.521 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:05.517+0000 7ff2d14d87c0 -1 osd.1 39 log_to_monitors true 2024-09-13T14:59:05.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:05.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:05.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:59:05.724 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T14:59:05.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:05.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:59:06.130 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 43 up_thru 38 down_at 40 last_clean_interval [38,39) [v2:127.0.0.1:6810/283103345,v1:127.0.0.1:6811/283103345] [v2:127.0.0.1:6812/283103345,v1:127.0.0.1:6813/283103345] exists,up efa940ef-fa1a-4793-9fef-ca428c78f418 2024-09-13T14:59:06.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:59:06.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:59:06.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:59:06.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T14:59:06.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T14:59:06.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T14:59:06.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T14:59:06.131 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T14:59:06.132 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T14:59:06.132 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T14:59:06.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T14:59:06.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T14:59:06.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T14:59:06.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T14:59:06.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T14:59:06.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T14:59:06.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T14:59:06.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T14:59:06.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T14:59:06.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T14:59:06.704 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T14:59:06.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T14:59:06.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:59:06.704 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T14:59:06.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149709 2024-09-13T14:59:06.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149709 2024-09-13T14:59:06.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149709' 2024-09-13T14:59:06.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T14:59:06.950 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T14:59:07.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=184683593730 2024-09-13T14:59:07.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 184683593730 2024-09-13T14:59:07.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149709 1-184683593730' 2024-09-13T14:59:07.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:59:07.098 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149709 2024-09-13T14:59:07.098 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:59:07.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T14:59:07.100 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149709 2024-09-13T14:59:07.100 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:59:07.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149709 2024-09-13T14:59:07.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149709' 2024-09-13T14:59:07.101 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 111669149709 2024-09-13T14:59:07.101 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:59:07.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149708 -lt 111669149709 2024-09-13T14:59:07.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T14:59:08.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T14:59:08.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T14:59:08.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149710 -lt 111669149709 2024-09-13T14:59:08.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T14:59:08.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-184683593730 2024-09-13T14:59:08.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T14:59:08.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T14:59:08.752 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-184683593730 2024-09-13T14:59:08.752 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T14:59:08.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=184683593730 2024-09-13T14:59:08.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 184683593730' 2024-09-13T14:59:08.754 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 184683593730 2024-09-13T14:59:08.754 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T14:59:09.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 184683593730 -lt 184683593730 2024-09-13T14:59:09.068 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T14:59:09.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:59:09.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:59:09.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T14:59:09.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T14:59:09.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T14:59:09.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T14:59:09.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T14:59:09.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T14:59:09.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T14:59:09.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T14:59:09.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T14:59:09.771 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T14:59:09.771 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T14:59:09.771 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T14:59:10.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T14:59:10.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T14:59:10.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T14:59:10.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T14:59:10.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T14:59:10.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T14:59:10.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:59:10.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:59:10.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:59:10.186 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:59:10.186 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:59:10.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:59:10.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:59:10.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:59:10.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:59:10.294 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:59:10.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:59:10.295 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:59:10.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:59:10.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:59:10.296 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:59:10.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:59:10.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:59:10.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:59:10.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:59:10.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:59:10.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:59:10.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:59:10.313 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:59:10.313 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:59:10.314 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:59:10.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:59:10.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:59:10.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:59:10.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T14:59:10.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T14:59:10.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T14:59:10.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T14:59:10.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T14:59:10.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T14:59:10.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T14:59:10.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T14:59:10.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T14:59:10.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T14:59:10.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T14:59:10.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T14:59:10.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T14:59:10.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T14:59:10.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T14:59:10.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T14:59:10.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T14:59:10.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T14:59:10.320 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T14:59:10.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:59:10.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T14:59:10.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T14:59:10.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T14:59:10.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T14:59:10.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T14:59:10.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T14:59:10.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T14:59:10.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:59:10.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:59:10.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T14:59:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T14:59:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T14:59:10.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T14:59:10.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T14:59:10.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:59:10.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:59:10.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T14:59:10.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T14:59:10.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T14:59:10.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T14:59:10.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T14:59:10.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-09-13T14:59:10.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-09-13T14:59:10.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-09-13T14:59:10.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-09-13T14:59:10.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-09-13T14:59:10.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-09-13T14:59:10.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-09-13T14:59:10.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T14:59:10.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T14:59:10.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T14:59:10.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T14:59:10.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T14:59:10.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T14:59:10.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T14:59:10.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:59:10.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:59:10.806 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:59:10.806 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:59:10.806 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:59:10.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:59:10.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T14:59:11.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T14:59:11.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T14:59:11.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:59:11.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:59:11.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T14:59:11.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T14:59:11.002 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:59:11.002 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:59:11.003 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:59:11.003 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:59:11.003 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:59:11.004 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:59:11.004 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:59:11.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:59:11.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T14:59:11.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T14:59:11.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T14:59:11.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T14:59:11.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T14:59:11.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T14:59:11.112 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T14:59:11.113 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T14:59:11.113 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T14:59:11.113 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T14:59:11.113 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:59:11.113 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:59:11.113 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T14:59:11.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T14:59:11.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T14:59:11.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-09-13T14:59:11.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T14:59:11.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T14:59:11.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T14:59:11.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T14:59:11.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T14:59:11.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T14:59:12.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T14:59:12.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:59:12.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:59:12.093 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:59:12.093 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:59:12.093 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:59:12.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:59:12.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:59:12.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T14:59:12.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-09-13T14:59:12.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-13T14:59:12.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-09-13T14:59:12.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:59:12.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:59:12.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T14:59:12.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:59:12.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T14:59:12.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:59:12.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:59:12.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:59:12.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:59:12.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T14:59:12.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T14:59:12.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:59:12.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:59:12.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:59:12.138 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:59:12.138 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:59:12.138 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:59:12.138 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:59:12.138 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:59:12.138 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:59:12.138 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:59:12.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:59:12.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:59:12.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:59:12.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:59:12.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:59:12.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:59:12.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:59:12.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:59:12.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:59:12.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:59:12.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:59:12.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:59:12.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T14:59:12.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:59:12.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=afd7d2c8-549f-44ac-ac39-8f66447cfc30 2024-09-13T14:59:12.140 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 afd7d2c8-549f-44ac-ac39-8f66447cfc30 2024-09-13T14:59:12.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 afd7d2c8-549f-44ac-ac39-8f66447cfc30' 2024-09-13T14:59:12.141 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:59:12.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBAU+RmfXB6CRAA3JQmC+opc+Ce29fQ0Lye1g== 2024-09-13T14:59:12.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBAU+RmfXB6CRAA3JQmC+opc+Ce29fQ0Lye1g=="}' 2024-09-13T14:59:12.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new afd7d2c8-549f-44ac-ac39-8f66447cfc30 -i td/osd-scrub-repair/0/new.json 2024-09-13T14:59:12.386 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:59:12.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T14:59:12.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAU+RmfXB6CRAA3JQmC+opc+Ce29fQ0Lye1g== --osd-uuid afd7d2c8-549f-44ac-ac39-8f66447cfc30 2024-09-13T14:59:12.432 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:12.429+0000 7faa166ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:59:12.445 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:12.441+0000 7faa166ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:59:12.446 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:12.441+0000 7faa166ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:59:12.446 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:12.445+0000 7faa166ed7c0 -1 bdev(0x55d149c6bc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T14:59:12.446 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:12.445+0000 7faa166ed7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T14:59:18.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T14:59:18.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:59:18.665 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T14:59:18.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T14:59:18.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:59:19.084 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T14:59:19.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T14:59:19.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:59:19.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:59:19.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:59:19.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:59:19.117 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:19.113+0000 7f24d37187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:59:19.133 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:19.129+0000 7f24d37187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:59:19.134 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:19.133+0000 7f24d37187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:59:19.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T14:59:19.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:59:19.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T14:59:19.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:59:19.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:59:19.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:19.392 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:59:19.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:59:19.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:19.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:19.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:20.429 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:20.425+0000 7f24d37187c0 -1 Falling back to public interface 2024-09-13T14:59:20.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:20.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:20.720 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:59:20.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:59:20.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:20.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:21.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:22.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:22.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:22.059 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:59:22.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:59:22.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:22.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:22.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:23.038 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:23.033+0000 7f24d37187c0 -1 osd.0 0 log_to_monitors true 2024-09-13T14:59:23.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:23.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:23.402 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:59:23.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:59:23.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:23.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:23.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:24.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:24.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:24.939 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:59:24.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:59:24.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:24.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:25.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:26.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:26.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:26.258 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:59:26.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:59:26.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:26.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:26.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:27.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:27.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:27.587 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T14:59:27.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T14:59:27.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:27.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:27.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:28.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:28.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:28.961 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T14:59:28.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T14:59:28.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:28.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:29.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:30.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:30.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:30.288 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T14:59:30.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T14:59:30.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:30.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:30.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:31.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:31.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:31.615 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T14:59:31.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T14:59:31.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:31.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:31.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:32.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:32.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:32.977 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T14:59:32.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T14:59:32.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:32.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:33.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:34.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:34.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:34.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T14:59:34.315 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T14:59:34.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:34.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:34.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:35.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:35.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:35.793 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T14:59:35.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T14:59:35.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:35.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:36.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:37.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:37.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:37.692 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T14:59:37.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T14:59:37.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:37.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:38.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:39.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:39.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:39.315 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T14:59:39.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T14:59:39.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:39.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:40.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:41.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:41.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:41.023 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T14:59:41.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T14:59:41.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:41.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:42.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:43.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:43.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:43.068 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T14:59:43.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T14:59:43.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:43.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:43.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:44.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:44.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:44.648 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T14:59:44.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T14:59:44.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:44.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T14:59:45.020 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2371922741,v1:127.0.0.1:6803/2371922741] [v2:127.0.0.1:6804/2371922741,v1:127.0.0.1:6805/2371922741] exists,up afd7d2c8-549f-44ac-ac39-8f66447cfc30 2024-09-13T14:59:45.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T14:59:45.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T14:59:45.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T14:59:45.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-13T14:59:45.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-09-13T14:59:45.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T14:59:45.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T14:59:45.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T14:59:45.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T14:59:45.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T14:59:45.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T14:59:45.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T14:59:45.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T14:59:45.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T14:59:45.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T14:59:45.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T14:59:45.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T14:59:45.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T14:59:45.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T14:59:45.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T14:59:45.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T14:59:45.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T14:59:45.024 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T14:59:45.024 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T14:59:45.024 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T14:59:45.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:59:45.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T14:59:45.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T14:59:45.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T14:59:45.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T14:59:45.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T14:59:45.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T14:59:45.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T14:59:45.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T14:59:45.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T14:59:45.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T14:59:45.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T14:59:45.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T14:59:45.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T14:59:45.027 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T14:59:45.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=82f257eb-8a0f-4fa6-be3d-e20fb6b47202 2024-09-13T14:59:45.028 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 82f257eb-8a0f-4fa6-be3d-e20fb6b47202 2024-09-13T14:59:45.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 82f257eb-8a0f-4fa6-be3d-e20fb6b47202' 2024-09-13T14:59:45.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T14:59:45.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBhU+Rmh5DxAhAAM0O0QuU6OK0ymg4ouw+kJA== 2024-09-13T14:59:45.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBhU+Rmh5DxAhAAM0O0QuU6OK0ymg4ouw+kJA=="}' 2024-09-13T14:59:45.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 82f257eb-8a0f-4fa6-be3d-e20fb6b47202 -i td/osd-scrub-repair/1/new.json 2024-09-13T14:59:45.426 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:59:45.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T14:59:45.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhU+Rmh5DxAhAAM0O0QuU6OK0ymg4ouw+kJA== --osd-uuid 82f257eb-8a0f-4fa6-be3d-e20fb6b47202 2024-09-13T14:59:45.481 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:45.480+0000 7f676faa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:59:45.506 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:45.504+0000 7f676faa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:59:45.507 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:45.504+0000 7f676faa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:59:45.507 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:45.504+0000 7f676faa37c0 -1 bdev(0x559a8aa0e800 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T14:59:45.507 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:45.504+0000 7f676faa37c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T14:59:51.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T14:59:51.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T14:59:51.345 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T14:59:51.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T14:59:51.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T14:59:51.754 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T14:59:51.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T14:59:51.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T14:59:51.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T14:59:51.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T14:59:51.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T14:59:51.786 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:51.788+0000 7f164074d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:59:51.821 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:51.820+0000 7f164074d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:59:51.822 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:51.824+0000 7f164074d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T14:59:52.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T14:59:52.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T14:59:52.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T14:59:52.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T14:59:52.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T14:59:52.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:52.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T14:59:52.092 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T14:59:52.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:52.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:59:52.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:53.121 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:53.120+0000 7f164074d7c0 -1 Falling back to public interface 2024-09-13T14:59:53.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:53.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:53.437 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T14:59:53.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T14:59:53.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:53.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:59:53.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:54.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:54.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:54.774 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T14:59:54.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T14:59:54.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:54.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:59:55.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:55.880 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T14:59:55.880+0000 7f164074d7c0 -1 osd.1 0 log_to_monitors true 2024-09-13T14:59:56.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:56.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:56.095 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T14:59:56.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T14:59:56.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:56.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:59:56.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:57.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:57.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:57.535 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T14:59:57.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T14:59:57.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:57.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:59:57.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T14:59:58.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T14:59:58.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T14:59:58.996 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T14:59:58.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T14:59:58.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T14:59:58.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T14:59:59.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:00.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:00.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:00.321 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:00:00.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:00:00.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:00.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:00:00.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:01.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:01.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:01.645 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:00:01.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:00:01.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:01.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:00:01.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:02.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:02.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:02.985 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:00:02.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:00:02.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:02.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:00:03.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:04.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:04.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:04.459 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:00:04.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:00:04.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:04.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:00:04.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:05.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:05.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:05.929 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:00:05.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:00:05.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:05.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:00:06.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:07.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:07.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:07.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:00:07.253 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:00:07.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:07.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:00:07.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:08.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:08.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:08.762 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:00:08.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:00:08.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:08.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:00:09.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:10.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:10.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:10.354 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:00:10.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:00:10.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:10.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:00:11.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:12.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:12.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:12.065 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:00:12.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:00:12.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:12.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:00:12.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:13.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:13.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:13.825 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:00:13.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:00:13.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:13.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:00:14.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:15.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:15.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:15.281 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:00:15.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:00:15.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:15.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:00:15.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:16.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:16.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:16.775 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:00:16.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:00:16.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:16.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:00:17.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:18.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:18.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:18.719 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:00:18.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:00:18.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:18.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:00:19.084 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1282191049,v1:127.0.0.1:6811/1282191049] [v2:127.0.0.1:6812/1282191049,v1:127.0.0.1:6813/1282191049] exists,up 82f257eb-8a0f-4fa6-be3d-e20fb6b47202 2024-09-13T15:00:19.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:00:19.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:00:19.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:00:19.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-13T15:00:19.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-09-13T15:00:19.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:00:19.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:00:19.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T15:00:19.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:00:19.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:00:19.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:00:19.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:00:19.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:00:19.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:00:19.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T15:00:19.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T15:00:19.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:00:19.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:00:19.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:00:19.089 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:00:19.089 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:00:19.089 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:00:19.089 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:00:19.089 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:00:19.089 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:00:19.089 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:00:19.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:00:19.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:00:19.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:00:19.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:00:19.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:00:19.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:00:19.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:00:19.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:00:19.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:00:19.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:00:19.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:00:19.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:00:19.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T15:00:19.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:00:19.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=630f71fb-2e86-4bdb-8ebe-08f3776fc180 2024-09-13T15:00:19.093 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 630f71fb-2e86-4bdb-8ebe-08f3776fc180 2024-09-13T15:00:19.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 630f71fb-2e86-4bdb-8ebe-08f3776fc180' 2024-09-13T15:00:19.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:00:19.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCDU+RmQXPIBhAAfvl7NY/8PJ/Wqxt+BaSOmg== 2024-09-13T15:00:19.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCDU+RmQXPIBhAAfvl7NY/8PJ/Wqxt+BaSOmg=="}' 2024-09-13T15:00:19.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 630f71fb-2e86-4bdb-8ebe-08f3776fc180 -i td/osd-scrub-repair/2/new.json 2024-09-13T15:00:19.466 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:00:19.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T15:00:19.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCDU+RmQXPIBhAAfvl7NY/8PJ/Wqxt+BaSOmg== --osd-uuid 630f71fb-2e86-4bdb-8ebe-08f3776fc180 2024-09-13T15:00:19.513 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:00:19.511+0000 7fde767337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:00:19.533 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:00:19.531+0000 7fde767337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:00:19.534 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:00:19.535+0000 7fde767337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:00:19.534 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:00:19.535+0000 7fde767337c0 -1 bdev(0x55bfadcadc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T15:00:19.535 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:00:19.535+0000 7fde767337c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T15:00:25.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T15:00:25.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:00:25.670 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T15:00:25.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T15:00:25.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:00:26.071 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T15:00:26.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T15:00:26.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:00:26.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:00:26.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:00:26.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:00:26.103 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:00:26.103+0000 7fb60c9d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:00:26.105 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:00:26.103+0000 7fb60c9d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:00:26.107 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:00:26.107+0000 7fb60c9d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:00:26.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T15:00:26.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:00:26.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:00:26.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:00:26.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:00:26.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:26.416 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:00:26.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:00:26.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:26.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:26.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:27.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:27.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:27.757 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:00:27.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:00:27.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:27.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:27.864 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:00:27.863+0000 7fb60c9d67c0 -1 Falling back to public interface 2024-09-13T15:00:28.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:29.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:29.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:29.084 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:00:29.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:00:29.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:29.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:29.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:30.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:30.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:30.441 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:00:30.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:00:30.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:30.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:30.549 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:00:30.547+0000 7fb60c9d67c0 -1 osd.2 0 log_to_monitors true 2024-09-13T15:00:30.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:31.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:31.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:31.768 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:00:31.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:00:31.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:31.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:32.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:33.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:33.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:33.090 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:00:33.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:00:33.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:33.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:33.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:34.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:34.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:34.435 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:00:34.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:00:34.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:34.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:34.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:35.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:35.766 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:00:35.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:35.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:00:35.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:35.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:36.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:37.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:37.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:37.095 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:00:37.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:00:37.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:37.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:37.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:38.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:38.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:38.423 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:00:38.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:00:38.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:38.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:38.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:39.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:39.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:39.745 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:00:39.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:00:39.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:39.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:40.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:41.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:41.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:41.084 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:00:41.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:00:41.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:41.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:41.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:42.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:42.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:42.408 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:00:42.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:00:42.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:42.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:43.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:44.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:44.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:44.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:00:44.127 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:00:44.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:44.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:44.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:45.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:45.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:45.472 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:00:45.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:00:45.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:45.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:45.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:46.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:46.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:46.975 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:00:46.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:00:46.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:46.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:47.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:48.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:48.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:48.792 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:00:48.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:00:48.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:48.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:49.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:50.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:50.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:50.223 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:00:50.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:00:50.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:50.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:50.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:51.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:51.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:51.636 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:00:51.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:00:51.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:51.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:52.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:53.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:53.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:53.039 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:00:53.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:00:53.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:53.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:53.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:00:54.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:00:54.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:00:54.369 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:00:54.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:00:54.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:00:54.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:00:54.699 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2091557081,v1:127.0.0.1:6819/2091557081] [v2:127.0.0.1:6820/2091557081,v1:127.0.0.1:6821/2091557081] exists,up 630f71fb-2e86-4bdb-8ebe-08f3776fc180 2024-09-13T15:00:54.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:00:54.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:00:54.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:00:54.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-09-13T15:00:54.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T15:00:55.019 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' does not exist 2024-09-13T15:00:55.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T15:00:55.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T15:00:55.553 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T15:00:55.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:00:56.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T15:00:58.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-09-13T15:00:58.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2024-09-13T15:00:59.163 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'foo' created 2024-09-13T15:00:59.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:01:00.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-09-13T15:01:00.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-13T15:01:00.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-13T15:01:00.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-13T15:01:00.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-13T15:01:00.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-09-13T15:01:00.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-13T15:01:00.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-13T15:01:01.434 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'ecpool' created 2024-09-13T15:01:01.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:01:02.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-13T15:01:02.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-13T15:01:02.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:01:02.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:01:02.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:01:02.459 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:01:02.460 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:01:02.460 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:01:02.460 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:01:02.461 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:01:02.461 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:01:02.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:01:02.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:01:02.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:01:02.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:01:02.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:01:02.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:01:03.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:01:03.062 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:01:03.062 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:01:03.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:01:03.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:01:03.062 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:01:03.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836498 2024-09-13T15:01:03.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836498 2024-09-13T15:01:03.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836498' 2024-09-13T15:01:03.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:01:03.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:01:03.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-13T15:01:03.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-13T15:01:03.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836498 1-42949672974' 2024-09-13T15:01:03.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:01:03.455 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:01:03.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509450 2024-09-13T15:01:03.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509450 2024-09-13T15:01:03.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836498 1-42949672974 2-64424509450' 2024-09-13T15:01:03.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:01:03.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:01:03.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836498 2024-09-13T15:01:03.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:01:03.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836498 2024-09-13T15:01:03.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:01:03.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836498 2024-09-13T15:01:03.600 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836498 2024-09-13T15:01:03.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-13T15:01:03.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:01:03.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836498 2024-09-13T15:01:03.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:01:04.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:01:04.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:01:05.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836498 -lt 21474836498 2024-09-13T15:01:05.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:01:05.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-13T15:01:05.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:01:05.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:01:05.247 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-13T15:01:05.247 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:01:05.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-13T15:01:05.249 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672974 2024-09-13T15:01:05.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-13T15:01:05.249 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:01:05.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672974 2024-09-13T15:01:05.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:01:05.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509450 2024-09-13T15:01:05.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:01:05.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:01:05.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509450 2024-09-13T15:01:05.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:01:05.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509450 2024-09-13T15:01:05.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-13T15:01:05.579 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509450 2024-09-13T15:01:05.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:01:05.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509450 2024-09-13T15:01:05.895 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:01:05.895 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:01:05.895 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:01:06.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:01:06.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:01:06.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:01:06.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:01:06.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:01:06.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:01:06.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:01:06.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:01:06.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:01:06.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:01:06.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:01:06.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:01:07.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:01:07.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:01:07.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:01:07.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-13T15:01:07.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-09-13T15:01:07.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:01:07.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:01:07.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:01:07.042 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:01:07.042 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:01:07.043 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:01:07.043 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:01:07.043 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:01:07.043 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:01:07.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:01:07.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:01:07.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:01:07.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:01:07.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:01:07.151 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:01:07.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:01:07.505 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:01:07.505 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:01:07.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:01:07.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:01:07.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:01:07.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836500 2024-09-13T15:01:07.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836500 2024-09-13T15:01:07.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500' 2024-09-13T15:01:07.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:01:07.647 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:01:07.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672976 2024-09-13T15:01:07.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672976 2024-09-13T15:01:07.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672976' 2024-09-13T15:01:07.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:01:07.779 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:01:07.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-13T15:01:07.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-13T15:01:07.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672976 2-64424509452' 2024-09-13T15:01:07.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:01:07.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836500 2024-09-13T15:01:07.920 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:01:07.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:01:07.921 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:01:07.921 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836500 2024-09-13T15:01:07.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836500 2024-09-13T15:01:07.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-13T15:01:07.923 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836500 2024-09-13T15:01:07.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:01:08.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836500 2024-09-13T15:01:08.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:01:09.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:01:09.240 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:01:09.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-13T15:01:09.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:01:09.563 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672976 2024-09-13T15:01:09.563 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:01:09.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:01:09.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672976 2024-09-13T15:01:09.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:01:09.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672976 2024-09-13T15:01:09.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-13T15:01:09.567 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672976 2024-09-13T15:01:09.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:01:09.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-13T15:01:09.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:01:09.892 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-13T15:01:09.892 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:01:09.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:01:09.894 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-13T15:01:09.894 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:01:09.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-13T15:01:09.896 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509452 2024-09-13T15:01:09.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-13T15:01:09.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:01:10.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-13T15:01:10.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:01:10.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:01:10.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:01:10.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:01:10.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:01:10.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:01:10.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:01:10.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:01:10.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:01:10.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:01:10.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:01:10.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:01:10.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:01:10.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:01:10.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:01:11.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:01:11.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:01:11.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:01:11.388 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-09-13T15:01:11.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-13T15:01:11.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-09-13T15:01:11.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-09-13T15:01:11.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:01:11.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:01:11.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-09-13T15:01:11.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:01:11.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:01:11.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:01:11.813 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:01:11.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:01:12.341 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:01:12.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:01:12.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:01:12.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:01:12.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-09-13T15:01:12.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-13T15:01:12.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-13T15:01:12.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-09-13T15:01:12.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-09-13T15:01:12.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:01:12.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:01:12.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:01:12.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:01:12.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:01:12.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:01:12.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:01:12.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:01:12.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:01:12.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:01:12.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:01:12.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:01:12.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:01:12.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:01:12.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:01:12.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:01:12.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:01:12.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:01:12.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:01:12.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:01:12.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:01:12.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:01:12.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:01:12.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:01:15.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:01:15.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:01:15.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:01:15.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:01:15.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:01:15.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:01:15.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:01:15.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:01:15.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:01:15.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:01:15.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:01:15.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:01:15.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:01:15.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:01:15.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:01:15.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:01:15.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:01:15.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:01:15.758 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:01:15.758 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:01:15.758 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:01:15.758 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:01:15.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:01:15.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:01:15.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:01:15.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:01:15.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:01:15.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:01:15.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:01:15.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:01:15.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:01:15.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:01:15.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:01:15.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:01:15.760 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:01:15.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:01:15.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:01:15.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:01:15.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:01:15.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:01:15.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:01:15.793 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:01:15.789+0000 7f3520dee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:01:15.797 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:01:15.793+0000 7f3520dee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:01:15.798 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:01:15.793+0000 7f3520dee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:01:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:01:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:01:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:01:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:01:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:01:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:01:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:01:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:01:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:01:16.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:01:16.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:01:17.298 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:01:17.293+0000 7f3520dee7c0 -1 Falling back to public interface 2024-09-13T15:01:17.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:01:17.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:01:17.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:01:17.439 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:01:17.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:01:17.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:01:17.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:01:18.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:01:18.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:01:18.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:01:18.758 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:01:18.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:01:18.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:01:19.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:01:20.033 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:01:20.029+0000 7f3520dee7c0 -1 osd.1 35 log_to_monitors true 2024-09-13T15:01:20.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:01:20.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:01:20.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:01:20.093 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:01:20.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:01:20.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:01:20.504 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 40 up_thru 30 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/1553606894,v1:127.0.0.1:6811/1553606894] [v2:127.0.0.1:6812/1553606894,v1:127.0.0.1:6813/1553606894] exists,up 82f257eb-8a0f-4fa6-be3d-e20fb6b47202 2024-09-13T15:01:20.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:01:20.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:01:20.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:01:20.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:01:20.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:01:20.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:01:20.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:01:20.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:01:20.506 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:01:20.506 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:01:20.506 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:01:20.506 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:01:20.506 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:01:20.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:01:20.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:01:20.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:01:20.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:01:20.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:01:20.613 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:01:21.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:01:21.021 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:01:21.021 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:01:21.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:01:21.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:01:21.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:01:21.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-13T15:01:21.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-13T15:01:21.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-13T15:01:21.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:01:21.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:01:21.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691842 2024-09-13T15:01:21.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691842 2024-09-13T15:01:21.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-171798691842' 2024-09-13T15:01:21.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:01:21.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:01:21.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509456 2024-09-13T15:01:21.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509456 2024-09-13T15:01:21.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-171798691842 2-64424509456' 2024-09-13T15:01:21.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:01:21.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-13T15:01:21.569 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:01:21.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:01:21.569 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-13T15:01:21.569 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:01:21.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-13T15:01:21.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-13T15:01:21.570 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836504 2024-09-13T15:01:21.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:01:21.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836504 2024-09-13T15:01:21.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:01:22.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:01:22.893 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:01:23.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-13T15:01:23.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:01:23.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-171798691842 2024-09-13T15:01:23.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:01:23.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:01:23.226 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-171798691842 2024-09-13T15:01:23.226 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:01:23.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691842 2024-09-13T15:01:23.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-09-13T15:01:23.228 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 171798691842 2024-09-13T15:01:23.228 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:01:23.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691842 -lt 171798691842 2024-09-13T15:01:23.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:01:23.556 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509456 2024-09-13T15:01:23.556 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:01:23.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:01:23.558 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509456 2024-09-13T15:01:23.558 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:01:23.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509456 2024-09-13T15:01:23.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-13T15:01:23.560 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509456 2024-09-13T15:01:23.560 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:01:23.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-13T15:01:23.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:01:23.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:01:23.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:01:24.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:01:24.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:01:24.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:01:24.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:01:24.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:01:24.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:01:24.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:01:24.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:01:24.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-13T15:01:24.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:01:24.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:01:24.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:01:24.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-13T15:01:24.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-13T15:01:24.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:01:24.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-13T15:01:24.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:01:24.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:01:25.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:01:25.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:01:25.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:01:25.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:01:25.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:01:25.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:01:25.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:01:25.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:01:25.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-13T15:01:25.424 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:01:25.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:01:25.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:01:25.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-13T15:01:25.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-13T15:01:25.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:01:25.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:01:25.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:01:25.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:01:25.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:01:25.798 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:01:25.798 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:01:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:01:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:01:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T15:01:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:01:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T15:01:26.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:01:26.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:01:26.390 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:01:26.390 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:01:26.390 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:01:26.390 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:01:26.390 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:01:26.390 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:01:26.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-13T15:01:26.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:01:26.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:01:26.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:01:27.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-13T15:01:27.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-13T15:01:27.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:01:27.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:01:27.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:01:27.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:01:27.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:01:27.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:01:27.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:01:27.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:01:27.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:01:27.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-13T15:01:27.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:01:27.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-13T15:01:27.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:01:27.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:01:27.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:01:27.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:01:27.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:01:27.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:01:27.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:01:27.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:01:28.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:01:28.224 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:01:28.224 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:01:28.224 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:01:28.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:01:28.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:01:28.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:01:28.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-13T15:01:28.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-09-13T15:01:28.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-09-13T15:01:28.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:01:28.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:01:28.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-09-13T15:01:28.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:01:28.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:01:28.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:01:29.136 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:01:29.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:01:29.563 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:01:29.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:01:29.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:01:29.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:01:29.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-09-13T15:01:29.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-13T15:01:29.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-13T15:01:29.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-09-13T15:01:29.873 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T15:01:29.873 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T15:01:29.873 INFO:tasks.workunit.client.0.smithi112.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000231116 s, 8.9 MB/s 2024-09-13T15:01:29.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:01:29.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:01:29.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:01:29.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:01:29.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:01:29.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:01:29.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:01:29.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:01:29.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:01:29.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:01:29.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:01:29.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:01:29.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:01:29.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:01:29.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:01:29.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:01:30.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:01:30.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:01:30.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:01:30.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:01:30.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:01:30.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:01:30.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:01:30.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:01:32.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:01:32.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:01:32.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:01:32.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:01:32.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:01:32.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:01:32.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:01:32.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:01:32.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:01:32.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:01:32.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:01:32.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:01:32.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:01:32.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:01:32.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:01:32.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:01:32.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:01:32.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:01:32.641 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:01:32.641 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:01:32.641 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:01:32.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:01:32.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:01:32.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:01:32.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:01:32.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:01:32.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:01:32.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:01:32.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:01:32.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:01:32.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:01:32.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:01:32.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:01:32.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:01:32.643 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:01:32.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:01:32.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:01:32.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:01:32.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:01:32.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:01:32.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:01:32.682 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:01:32.677+0000 7fa16db877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:01:32.685 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:01:32.681+0000 7fa16db877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:01:32.687 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:01:32.685+0000 7fa16db877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:01:32.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:01:32.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:01:32.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:01:32.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:01:32.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:01:32.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:01:32.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:01:32.977 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:01:32.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:01:32.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:01:33.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:01:33.970 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:01:33.965+0000 7fa16db877c0 -1 Falling back to public interface 2024-09-13T15:01:34.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:01:34.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:01:34.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:01:34.326 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:01:34.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:01:34.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:01:34.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:01:35.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:01:35.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:01:35.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:01:35.658 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:01:35.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:01:35.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:01:35.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:01:36.613 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:01:36.609+0000 7fa16db877c0 -1 osd.0 44 log_to_monitors true 2024-09-13T15:01:36.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:01:36.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:01:36.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:01:36.996 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:01:36.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:01:36.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:01:37.553 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 49 up_thru 40 down_at 45 last_clean_interval [5,44) [v2:127.0.0.1:6802/2315626058,v1:127.0.0.1:6803/2315626058] [v2:127.0.0.1:6804/2315626058,v1:127.0.0.1:6805/2315626058] exists,up afd7d2c8-549f-44ac-ac39-8f66447cfc30 2024-09-13T15:01:37.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:01:37.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:01:37.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:01:37.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:01:37.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:01:37.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:01:37.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:01:37.555 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:01:37.555 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:01:37.555 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:01:37.555 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:01:37.555 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:01:37.556 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:01:37.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:01:37.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:01:37.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:01:37.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:01:37.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:01:37.678 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:01:38.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:01:38.117 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:01:38.117 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:01:38.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:01:38.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:01:38.117 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:01:38.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=210453397506 2024-09-13T15:01:38.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 210453397506 2024-09-13T15:01:38.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397506' 2024-09-13T15:01:38.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:01:38.313 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:01:38.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691846 2024-09-13T15:01:38.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691846 2024-09-13T15:01:38.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397506 1-171798691846' 2024-09-13T15:01:38.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:01:38.448 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:01:38.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509460 2024-09-13T15:01:38.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509460 2024-09-13T15:01:38.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397506 1-171798691846 2-64424509460' 2024-09-13T15:01:38.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:01:38.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-210453397506 2024-09-13T15:01:38.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:01:38.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:01:38.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-210453397506 2024-09-13T15:01:38.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:01:38.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=210453397506 2024-09-13T15:01:38.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 210453397506' 2024-09-13T15:01:38.591 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 210453397506 2024-09-13T15:01:38.591 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:01:38.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 210453397506 2024-09-13T15:01:38.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:01:39.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:01:39.920 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:01:40.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397506 -lt 210453397506 2024-09-13T15:01:40.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:01:40.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-171798691846 2024-09-13T15:01:40.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:01:40.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:01:40.263 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-171798691846 2024-09-13T15:01:40.263 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:01:40.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691846 2024-09-13T15:01:40.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 171798691846' 2024-09-13T15:01:40.264 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 171798691846 2024-09-13T15:01:40.265 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:01:40.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691846 -lt 171798691846 2024-09-13T15:01:40.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:01:40.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509460 2024-09-13T15:01:40.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:01:40.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:01:40.609 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509460 2024-09-13T15:01:40.609 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:01:40.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509460 2024-09-13T15:01:40.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-13T15:01:40.610 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509460 2024-09-13T15:01:40.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:01:40.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509461 -lt 64424509460 2024-09-13T15:01:40.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:01:40.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:01:40.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:01:41.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:01:41.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:01:41.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:01:41.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:01:41.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:01:41.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:01:41.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:01:41.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:01:41.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:01:41.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:01:41.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:01:41.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:01:42.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:01:42.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:01:42.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:01:42.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-13T15:01:42.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-09-13T15:01:42.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-09-13T15:01:42.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:01:42.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:01:42.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-09-13T15:01:42.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:01:42.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:01:42.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:01:42.542 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:01:42.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:01:43.095 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:01:43.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:01:43.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:01:43.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:01:43.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-09-13T15:01:43.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-13T15:01:43.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-13T15:01:43.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-09-13T15:01:43.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:01:43.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:01:43.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:01:43.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:01:43.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-09-13T15:01:43.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:01:43.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:01:43.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:01:43.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:01:43.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:01:43.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:01:43.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:01:43.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:01:43.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:01:43.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:01:43.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:01:43.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-09-13T15:01:43.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:01:43.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:01:43.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:01:43.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:01:43.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:01:43.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-09-13T15:01:45.685 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-09-13T15:01:46.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:01:46.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:01:46.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:01:46.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:01:46.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:01:46.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:01:46.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:01:46.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:01:46.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:01:46.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:01:46.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:01:46.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:01:46.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:01:46.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:01:46.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:01:46.232 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:01:46.232 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:01:46.232 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:01:46.233 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:01:46.233 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:01:46.233 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:01:46.233 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:01:46.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:01:46.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:01:46.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:01:46.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:01:46.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:01:46.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:01:46.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:01:46.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:01:46.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:01:46.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:01:46.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:01:46.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:01:46.234 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:01:46.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:01:46.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:01:46.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:01:46.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:01:46.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:01:46.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:01:46.269 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:01:46.264+0000 7fb66ad037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:01:46.272 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:01:46.268+0000 7fb66ad037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:01:46.273 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:01:46.268+0000 7fb66ad037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:01:46.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:01:46.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:01:46.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:01:46.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:01:46.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:01:46.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:01:46.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:01:46.567 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:01:46.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:01:46.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:01:46.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:01:47.782 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:01:47.776+0000 7fb66ad037c0 -1 Falling back to public interface 2024-09-13T15:01:47.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:01:47.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:01:47.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:01:47.905 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:01:47.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:01:47.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:01:48.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:01:49.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:01:49.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:01:49.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:01:49.224 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:01:49.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:01:49.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:01:49.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:01:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:01:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:01:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:01:50.543 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:01:50.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:01:50.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:01:50.627 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:01:50.620+0000 7fb66ad037c0 -1 osd.1 53 log_to_monitors true 2024-09-13T15:01:50.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:01:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:01:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:01:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:01:51.991 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:01:51.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:01:51.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:01:52.366 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 54 last_clean_interval [40,53) [v2:127.0.0.1:6810/4237631139,v1:127.0.0.1:6811/4237631139] [v2:127.0.0.1:6812/4237631139,v1:127.0.0.1:6813/4237631139] exists,up 82f257eb-8a0f-4fa6-be3d-e20fb6b47202 2024-09-13T15:01:52.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:01:52.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:01:52.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:01:52.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:01:52.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:01:52.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:01:52.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:01:52.368 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:01:52.368 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:01:52.368 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:01:52.368 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:01:52.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:01:52.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:01:52.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:01:52.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:01:52.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:01:52.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:01:52.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:01:52.479 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:01:52.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:01:52.814 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:01:52.814 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:01:52.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:01:52.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:01:52.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:01:52.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=210453397510 2024-09-13T15:01:52.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 210453397510 2024-09-13T15:01:52.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397510' 2024-09-13T15:01:52.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:01:52.996 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:01:53.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=249108103170 2024-09-13T15:01:53.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 249108103170 2024-09-13T15:01:53.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397510 1-249108103170' 2024-09-13T15:01:53.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:01:53.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:01:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509464 2024-09-13T15:01:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509464 2024-09-13T15:01:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-210453397510 1-249108103170 2-64424509464' 2024-09-13T15:01:53.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:01:53.278 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-210453397510 2024-09-13T15:01:53.278 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:01:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:01:53.280 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-210453397510 2024-09-13T15:01:53.280 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:01:53.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=210453397510 2024-09-13T15:01:53.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 210453397510' 2024-09-13T15:01:53.281 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 210453397510 2024-09-13T15:01:53.281 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:01:53.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397509 -lt 210453397510 2024-09-13T15:01:53.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:01:54.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:01:54.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:01:54.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397509 -lt 210453397510 2024-09-13T15:01:54.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:01:55.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T15:01:55.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:01:56.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397510 -lt 210453397510 2024-09-13T15:01:56.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:01:56.269 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-249108103170 2024-09-13T15:01:56.269 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:01:56.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:01:56.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-249108103170 2024-09-13T15:01:56.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:01:56.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=249108103170 2024-09-13T15:01:56.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 249108103170' 2024-09-13T15:01:56.274 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 249108103170 2024-09-13T15:01:56.274 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:01:56.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103170 -lt 249108103170 2024-09-13T15:01:56.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:01:56.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509464 2024-09-13T15:01:56.613 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:01:56.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:01:56.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509464 2024-09-13T15:01:56.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:01:56.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509464 2024-09-13T15:01:56.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-09-13T15:01:56.618 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509464 2024-09-13T15:01:56.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:01:56.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509465 -lt 64424509464 2024-09-13T15:01:56.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:01:56.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:01:56.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:01:57.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:01:57.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:01:57.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:01:57.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:01:57.342 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:01:57.342 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:01:57.342 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:01:57.342 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:01:57.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:01:57.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:01:57.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:01:57.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:01:58.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:01:58.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:01:58.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:01:58.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-13T15:01:58.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-09-13T15:01:58.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-09-13T15:01:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:01:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:01:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-09-13T15:01:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:01:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:01:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:01:58.493 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:01:58.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:01:58.961 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:01:58.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:01:58.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:01:58.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:01:59.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-09-13T15:01:59.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-13T15:01:59.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-13T15:01:59.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-09-13T15:01:59.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-09-13T15:01:59.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-09-13T15:01:59.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-13T15:01:59.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:01:59.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:01:59.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:01:59.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:01:59.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-13T15:01:59.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:01:59.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:01:59.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:01:59.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:01:59.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:01:59.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:01:59.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:01:59.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:01:59.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:01:59.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:01:59.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:01:59.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-13T15:01:59.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:01:59.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:01:59.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:01:59.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:01:59.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:01:59.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-13T15:02:02.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:02:02.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:02:02.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:02:02.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:02:02.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:02:02.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:02:02.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:02:02.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:02:02.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:02:02.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:02:02.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:02:02.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:02:02.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:02:02.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:02:02.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:02:02.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:02:02.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:02:02.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:02:02.423 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:02:02.423 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:02:02.423 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:02:02.424 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:02:02.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:02:02.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:02:02.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:02:02.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:02:02.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:02:02.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:02:02.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:02:02.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:02:02.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:02:02.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:02:02.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:02:02.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:02:02.425 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:02:02.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:02:02.426 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:02:02.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:02:02.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:02:02.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:02:02.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:02:02.459 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:02.460+0000 7f387a4d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:02:02.462 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:02.460+0000 7f387a4d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:02:02.463 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:02.464+0000 7f387a4d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:02:02.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:02:02.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:02:02.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:02:02.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:02:02.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:02:02.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:02.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:02:02.757 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:02:02.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:02.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:02:03.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:02:03.498 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:03.496+0000 7f387a4d77c0 -1 Falling back to public interface 2024-09-13T15:02:04.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:02:04.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:04.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:02:04.130 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:02:04.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:04.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:02:04.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:02:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:02:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:02:05.457 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:02:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:05.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:02:05.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:02:06.186 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:06.184+0000 7f387a4d77c0 -1 osd.0 62 log_to_monitors true 2024-09-13T15:02:06.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:02:06.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:06.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:02:06.788 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:02:06.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:06.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:02:07.273 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 67 up_thru 67 down_at 63 last_clean_interval [49,62) [v2:127.0.0.1:6802/3342171482,v1:127.0.0.1:6803/3342171482] [v2:127.0.0.1:6804/3342171482,v1:127.0.0.1:6805/3342171482] exists,up afd7d2c8-549f-44ac-ac39-8f66447cfc30 2024-09-13T15:02:07.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:02:07.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:02:07.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:02:07.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:02:07.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:02:07.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:02:07.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:02:07.274 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:02:07.274 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:02:07.274 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:02:07.275 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:02:07.275 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:02:07.275 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:02:07.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:02:07.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:02:07.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:02:07.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:02:07.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:02:07.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:02:07.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:02:07.927 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:02:07.927 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:02:07.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:02:07.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:02:07.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:02:08.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808834 2024-09-13T15:02:08.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808834 2024-09-13T15:02:08.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808834' 2024-09-13T15:02:08.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:02:08.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:02:08.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=249108103174 2024-09-13T15:02:08.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 249108103174 2024-09-13T15:02:08.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808834 1-249108103174' 2024-09-13T15:02:08.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:02:08.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:02:08.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509468 2024-09-13T15:02:08.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509468 2024-09-13T15:02:08.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808834 1-249108103174 2-64424509468' 2024-09-13T15:02:08.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:02:08.396 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808834 2024-09-13T15:02:08.396 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:02:08.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:02:08.397 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808834 2024-09-13T15:02:08.397 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:02:08.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808834 2024-09-13T15:02:08.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808834' 2024-09-13T15:02:08.398 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 287762808834 2024-09-13T15:02:08.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:02:08.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 287762808834 2024-09-13T15:02:08.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:02:09.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:02:09.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:02:10.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808834 -lt 287762808834 2024-09-13T15:02:10.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:02:10.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-249108103174 2024-09-13T15:02:10.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:02:10.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:02:10.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-249108103174 2024-09-13T15:02:10.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:02:10.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=249108103174 2024-09-13T15:02:10.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 249108103174' 2024-09-13T15:02:10.023 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 249108103174 2024-09-13T15:02:10.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:02:10.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103174 -lt 249108103174 2024-09-13T15:02:10.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:02:10.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:02:10.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509468 2024-09-13T15:02:10.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:02:10.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509468 2024-09-13T15:02:10.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:02:10.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509468 2024-09-13T15:02:10.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-09-13T15:02:10.382 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509468 2024-09-13T15:02:10.382 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:02:10.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509468 -lt 64424509468 2024-09-13T15:02:10.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:02:10.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:02:10.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:02:11.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:02:11.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:02:11.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:02:11.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:02:11.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:02:11.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:02:11.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:02:11.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:02:11.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:02:11.459 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:02:11.459 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:02:11.460 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:02:11.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:02:11.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:02:11.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:02:11.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-13T15:02:11.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:02:11.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:02:11.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-13T15:02:11.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:02:11.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-13T15:02:11.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:02:11.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:02:11.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-13T15:02:11.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:02:11.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-13T15:02:11.858 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:02:11.858 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:02:11.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:02:11.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:02:11.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:02:12.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:02:12.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-13T15:02:12.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:02:12.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:02:12.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-13T15:02:12.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:02:12.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:02:12.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-13T15:02:14.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-13T15:02:14.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:02:14.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:02:14.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T15:02:14.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:02:14.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:02:14.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:02:14.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:02:14.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:02:14.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:02:14.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T15:02:14.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T15:02:14.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:02:14.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:02:14.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:02:14.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:02:14.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:02:14.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:02:14.497 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:02:14.497 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:02:14.497 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:02:14.498 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:02:14.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:02:14.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:02:14.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:02:14.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:02:14.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:02:14.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:02:14.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:02:14.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:02:14.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:02:14.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:02:14.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T15:02:14.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T15:02:14.500 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.2 2024-09-13T15:02:14.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:02:14.500 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-13T15:02:14.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T15:02:14.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:02:14.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:02:14.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:02:14.539 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:14.536+0000 7f44701a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:02:14.542 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:14.540+0000 7f44701a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:02:14.544 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:14.544+0000 7f44701a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:02:14.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T15:02:14.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:02:14.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:02:14.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:02:14.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:02:14.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:14.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:02:14.829 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:02:14.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:14.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:02:15.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:02:16.002 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:16.000+0000 7f44701a77c0 -1 Falling back to public interface 2024-09-13T15:02:16.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:02:16.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:16.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:02:16.197 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:02:16.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:16.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:02:16.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:02:17.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:02:17.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:17.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:02:17.533 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:02:17.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:17.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:02:17.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:02:18.717 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:18.715+0000 7f44701a77c0 -1 osd.2 69 log_to_monitors true 2024-09-13T15:02:18.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:02:18.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:18.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:02:18.867 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:02:18.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:18.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:02:19.339 INFO:tasks.workunit.client.0.smithi112.stderr:osd.2 up in weight 1 up_from 74 up_thru 67 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/992668758,v1:127.0.0.1:6819/992668758] [v2:127.0.0.1:6820/992668758,v1:127.0.0.1:6821/992668758] exists,up 630f71fb-2e86-4bdb-8ebe-08f3776fc180 2024-09-13T15:02:19.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:02:19.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:02:19.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:02:19.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:02:19.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:02:19.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:02:19.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:02:19.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:02:19.341 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:02:19.341 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:02:19.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:02:19.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:02:19.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:02:19.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:02:19.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:02:19.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:02:19.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:02:19.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:02:19.445 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:02:19.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:02:19.795 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:02:19.795 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:02:19.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:02:19.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:02:19.796 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:02:19.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808837 2024-09-13T15:02:19.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808837 2024-09-13T15:02:19.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808837' 2024-09-13T15:02:19.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:02:19.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:02:20.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=249108103177 2024-09-13T15:02:20.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 249108103177 2024-09-13T15:02:20.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808837 1-249108103177' 2024-09-13T15:02:20.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:02:20.182 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:02:20.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=317827579906 2024-09-13T15:02:20.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 317827579906 2024-09-13T15:02:20.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808837 1-249108103177 2-317827579906' 2024-09-13T15:02:20.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:02:20.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808837 2024-09-13T15:02:20.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:02:20.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:02:20.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808837 2024-09-13T15:02:20.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:02:20.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808837 2024-09-13T15:02:20.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808837' 2024-09-13T15:02:20.363 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 287762808837 2024-09-13T15:02:20.363 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:02:20.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808836 -lt 287762808837 2024-09-13T15:02:20.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:02:21.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:02:21.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:02:22.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808837 -lt 287762808837 2024-09-13T15:02:22.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:02:22.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-249108103177 2024-09-13T15:02:22.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:02:22.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:02:22.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-249108103177 2024-09-13T15:02:22.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:02:22.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=249108103177 2024-09-13T15:02:22.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 249108103177' 2024-09-13T15:02:22.075 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 249108103177 2024-09-13T15:02:22.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:02:22.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103177 -lt 249108103177 2024-09-13T15:02:22.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:02:22.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-317827579906 2024-09-13T15:02:22.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:02:22.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:02:22.419 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-317827579906 2024-09-13T15:02:22.419 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:02:22.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=317827579906 2024-09-13T15:02:22.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-09-13T15:02:22.421 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 317827579906 2024-09-13T15:02:22.421 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:02:22.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 317827579906 -lt 317827579906 2024-09-13T15:02:22.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:02:22.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:02:22.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:02:23.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:02:23.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:02:23.202 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:02:23.202 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:02:23.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:02:23.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:02:23.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:02:23.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:02:23.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:02:23.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:02:23.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:02:23.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:02:23.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:02:23.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:02:23.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:02:23.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-09-13T15:02:23.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-13T15:02:23.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:02:23.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:02:23.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-13T15:02:23.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:02:23.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-13T15:02:23.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:02:23.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:02:23.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-13T15:02:23.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:02:23.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-13T15:02:23.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:02:23.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:02:23.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:02:23.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:02:23.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:02:24.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:02:24.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-13T15:02:24.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:02:24.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:02:24.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-13T15:02:24.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:02:24.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:02:24.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-13T15:02:26.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-13T15:02:26.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:02:26.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:02:26.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T15:02:26.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:02:26.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:02:26.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:02:26.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:02:26.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:02:26.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:02:26.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T15:02:26.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T15:02:26.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:02:26.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:02:26.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:02:26.650 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:02:26.651 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:02:26.651 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:02:26.651 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:02:26.651 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:02:26.651 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:02:26.652 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:02:26.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:02:26.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:02:26.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:02:26.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:02:26.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:02:26.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:02:26.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:02:26.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:02:26.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:02:26.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:02:26.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T15:02:26.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T15:02:26.653 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.2 2024-09-13T15:02:26.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:02:26.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-13T15:02:26.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T15:02:26.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:02:26.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:02:26.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:02:26.688 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:26.687+0000 7f1400c4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:02:26.690 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:26.687+0000 7f1400c4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:02:26.692 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:26.691+0000 7f1400c4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:02:26.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T15:02:26.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:02:26.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:02:26.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:02:26.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:02:26.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:26.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:02:26.989 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:02:26.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:26.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:02:27.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:02:28.174 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:28.171+0000 7f1400c4b7c0 -1 Falling back to public interface 2024-09-13T15:02:28.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:02:28.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:28.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:02:28.360 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:02:28.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:28.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:02:28.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:02:29.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:02:29.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:29.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:02:29.694 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:02:29.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:29.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:02:30.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:02:30.825 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:30.823+0000 7f1400c4b7c0 -1 osd.2 76 log_to_monitors true 2024-09-13T15:02:31.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:02:31.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:31.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:02:31.031 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:02:31.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:31.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:02:31.452 INFO:tasks.workunit.client.0.smithi112.stderr:osd.2 up in weight 1 up_from 81 up_thru 74 down_at 77 last_clean_interval [74,76) [v2:127.0.0.1:6818/2731954594,v1:127.0.0.1:6819/2731954594] [v2:127.0.0.1:6820/2731954594,v1:127.0.0.1:6821/2731954594] exists,up 630f71fb-2e86-4bdb-8ebe-08f3776fc180 2024-09-13T15:02:31.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:02:31.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:02:31.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:02:31.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:02:31.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:02:31.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:02:31.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:02:31.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:02:31.454 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:02:31.454 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:02:31.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:02:31.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:02:31.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:02:31.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:02:31.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:02:31.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:02:31.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:02:31.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:02:31.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:02:31.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:02:31.890 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:02:31.890 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:02:31.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:02:31.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:02:31.891 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:02:32.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808841 2024-09-13T15:02:32.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808841 2024-09-13T15:02:32.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808841' 2024-09-13T15:02:32.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:02:32.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:02:32.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=249108103181 2024-09-13T15:02:32.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 249108103181 2024-09-13T15:02:32.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808841 1-249108103181' 2024-09-13T15:02:32.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:02:32.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:02:32.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350978 2024-09-13T15:02:32.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350978 2024-09-13T15:02:32.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808841 1-249108103181 2-347892350978' 2024-09-13T15:02:32.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:02:32.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808841 2024-09-13T15:02:32.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:02:32.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:02:32.371 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808841 2024-09-13T15:02:32.371 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:02:32.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808841 2024-09-13T15:02:32.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808841' 2024-09-13T15:02:32.372 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 287762808841 2024-09-13T15:02:32.373 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:02:32.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808839 -lt 287762808841 2024-09-13T15:02:32.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:02:33.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:02:33.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:02:34.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808841 -lt 287762808841 2024-09-13T15:02:34.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:02:34.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-249108103181 2024-09-13T15:02:34.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:02:34.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:02:34.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-249108103181 2024-09-13T15:02:34.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:02:34.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=249108103181 2024-09-13T15:02:34.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 249108103181' 2024-09-13T15:02:34.025 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 249108103181 2024-09-13T15:02:34.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:02:34.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 249108103181 -lt 249108103181 2024-09-13T15:02:34.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:02:34.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-347892350978 2024-09-13T15:02:34.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:02:34.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:02:34.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-347892350978 2024-09-13T15:02:34.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:02:34.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350978 2024-09-13T15:02:34.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 347892350978' 2024-09-13T15:02:34.351 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 347892350978 2024-09-13T15:02:34.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:02:34.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350978 -lt 347892350978 2024-09-13T15:02:34.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:02:34.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:02:34.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:02:35.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:02:35.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:02:35.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:02:35.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:02:35.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:02:35.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:02:35.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:02:35.086 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:02:35.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-13T15:02:35.426 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:02:35.427 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:02:35.427 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:02:35.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-13T15:02:35.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-13T15:02:35.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:02:35.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-13T15:02:35.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:02:35.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:02:35.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:02:35.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:02:35.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:02:35.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:02:35.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:02:35.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:02:35.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:02:35.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:02:36.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-13T15:02:36.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:02:36.219 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:02:36.219 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:02:36.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-13T15:02:36.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-13T15:02:36.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:02:36.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:02:36.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:02:36.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:02:36.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:02:36.613 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:02:36.613 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:02:37.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:02:37.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:02:37.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T15:02:37.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:02:37.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T15:02:37.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:02:37.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:02:37.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:02:37.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:02:37.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:02:37.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:02:37.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:02:37.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:02:37.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:02:37.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:02:37.546 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:02:37.546 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:02:37.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:02:37.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:02:37.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:02:37.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-09-13T15:02:37.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-13T15:02:37.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-09-13T15:02:37.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-09-13T15:02:37.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:02:37.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:02:37.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-09-13T15:02:37.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:02:37.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:02:37.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:02:38.371 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:02:38.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:02:38.865 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:02:38.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:02:38.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:02:38.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:02:39.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-09-13T15:02:39.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-13T15:02:39.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-13T15:02:39.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-09-13T15:02:39.108 INFO:tasks.workunit.client.0.smithi112.stderr:2+0 records in 2024-09-13T15:02:39.108 INFO:tasks.workunit.client.0.smithi112.stderr:2+0 records out 2024-09-13T15:02:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000170932 s, 24.0 MB/s 2024-09-13T15:02:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:02:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:02:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:02:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:02:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:02:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:02:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:02:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:02:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:02:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:02:39.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:02:39.110 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:02:39.110 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:02:39.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:02:39.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:02:39.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:02:39.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:02:39.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:02:39.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:02:39.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:02:39.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:02:39.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:02:39.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:02:39.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:02:41.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:02:41.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:02:41.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:02:41.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:02:41.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:02:41.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:02:41.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:02:41.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:02:41.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:02:41.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:02:41.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:02:41.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:02:41.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:02:41.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:02:41.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:02:41.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:02:41.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:02:41.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:02:41.823 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:02:41.823 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:02:41.823 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:02:41.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:02:41.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:02:41.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:02:41.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:02:41.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:02:41.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:02:41.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:02:41.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:02:41.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:02:41.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:02:41.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:02:41.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:02:41.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:02:41.825 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:02:41.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:02:41.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:02:41.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:02:41.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:02:41.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:02:41.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:02:41.859 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:41.859+0000 7f86c34247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:02:41.862 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:41.859+0000 7f86c34247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:02:41.864 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:41.863+0000 7f86c34247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:02:42.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:02:42.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:02:42.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:02:42.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:02:42.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:02:42.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:42.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:02:42.155 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:02:42.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:42.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:02:42.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:02:42.942 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:42.939+0000 7f86c34247c0 -1 Falling back to public interface 2024-09-13T15:02:43.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:02:43.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:43.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:02:43.518 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:02:43.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:43.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:02:43.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:02:44.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:02:44.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:44.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:02:44.858 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:02:44.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:44.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:02:45.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:02:45.708 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:45.707+0000 7f86c34247c0 -1 osd.1 85 log_to_monitors true 2024-09-13T15:02:46.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:02:46.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:46.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:02:46.187 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:02:46.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:46.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:02:46.674 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 90 up_thru 81 down_at 86 last_clean_interval [58,85) [v2:127.0.0.1:6810/2757741408,v1:127.0.0.1:6811/2757741408] [v2:127.0.0.1:6812/2757741408,v1:127.0.0.1:6813/2757741408] exists,up 82f257eb-8a0f-4fa6-be3d-e20fb6b47202 2024-09-13T15:02:46.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:02:46.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:02:46.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:02:46.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:02:46.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:02:46.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:02:46.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:02:46.675 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:02:46.675 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:02:46.675 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:02:46.676 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:02:46.676 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:02:46.676 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:02:46.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:02:46.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:02:46.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:02:46.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:02:46.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:02:46.782 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:02:47.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:02:47.209 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:02:47.209 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:02:47.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:02:47.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:02:47.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:02:47.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808845 2024-09-13T15:02:47.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808845 2024-09-13T15:02:47.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808845' 2024-09-13T15:02:47.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:02:47.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:02:47.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=386547056642 2024-09-13T15:02:47.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 386547056642 2024-09-13T15:02:47.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808845 1-386547056642' 2024-09-13T15:02:47.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:02:47.553 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:02:47.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350982 2024-09-13T15:02:47.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350982 2024-09-13T15:02:47.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808845 1-386547056642 2-347892350982' 2024-09-13T15:02:47.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:02:47.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808845 2024-09-13T15:02:47.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:02:47.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:02:47.697 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808845 2024-09-13T15:02:47.697 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:02:47.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808845 2024-09-13T15:02:47.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808845' 2024-09-13T15:02:47.698 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 287762808845 2024-09-13T15:02:47.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:02:48.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808844 -lt 287762808845 2024-09-13T15:02:48.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:02:49.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:02:49.007 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:02:49.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808845 -lt 287762808845 2024-09-13T15:02:49.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:02:49.342 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-386547056642 2024-09-13T15:02:49.342 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:02:49.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:02:49.344 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-386547056642 2024-09-13T15:02:49.344 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:02:49.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=386547056642 2024-09-13T15:02:49.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2024-09-13T15:02:49.347 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 386547056642 2024-09-13T15:02:49.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:02:49.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 386547056642 -lt 386547056642 2024-09-13T15:02:49.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:02:49.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-347892350982 2024-09-13T15:02:49.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:02:49.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:02:49.675 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-347892350982 2024-09-13T15:02:49.675 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:02:49.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350982 2024-09-13T15:02:49.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 347892350982' 2024-09-13T15:02:49.677 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 347892350982 2024-09-13T15:02:49.677 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:02:49.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350982 -lt 347892350982 2024-09-13T15:02:49.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:02:49.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:02:49.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:02:50.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:02:50.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:02:50.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:02:50.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:02:50.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:02:50.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:02:50.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:02:50.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:02:50.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T15:02:50.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:02:50.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:02:50.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:02:51.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-13T15:02:51.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-13T15:02:51.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:02:51.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-13T15:02:51.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:02:51.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:02:51.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:02:51.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:02:51.206 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:02:51.206 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:02:51.207 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:02:51.207 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:02:51.207 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:02:51.207 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:02:51.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T15:02:51.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:02:51.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:02:51.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:02:51.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-13T15:02:51.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-13T15:02:51.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:02:51.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:02:51.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:02:51.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:02:51.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:02:51.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:02:51.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:02:52.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:02:52.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:02:52.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T15:02:52.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:02:52.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T15:02:52.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:02:52.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:02:52.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:02:52.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:02:52.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:02:52.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:02:52.542 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:02:52.542 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:02:52.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:02:52.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:02:52.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:02:52.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:02:53.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:02:53.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:02:53.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:02:53.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-13T15:02:53.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-09-13T15:02:53.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-09-13T15:02:53.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:02:53.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:02:53.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-09-13T15:02:53.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:02:53.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:02:53.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:02:53.687 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:02:53.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:02:54.181 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:02:54.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:02:54.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:02:54.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:02:54.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-09-13T15:02:54.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-13T15:02:54.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-13T15:02:54.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-13T15:02:54.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:02:54.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:02:54.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:02:54.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:02:54.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-13T15:02:54.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:02:54.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:02:54.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:02:54.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:02:54.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:02:54.382 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:02:54.382 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:02:54.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:02:54.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:02:54.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:02:54.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:02:54.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-13T15:02:54.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:02:54.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:02:54.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:02:54.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:02:54.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:02:54.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-09-13T15:02:56.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:02:56.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:02:56.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:02:56.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:02:56.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:02:56.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:02:56.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:02:56.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:02:56.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:02:56.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:02:56.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:02:56.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:02:56.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:02:56.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:02:56.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:02:56.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:02:56.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:02:56.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:02:56.755 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:02:56.755 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:02:56.755 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:02:56.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:02:56.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:02:56.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:02:56.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:02:56.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:02:56.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:02:56.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:02:56.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:02:56.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:02:56.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:02:56.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:02:56.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:02:56.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:02:56.757 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:02:56.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:02:56.758 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:02:56.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:02:56.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:02:56.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:02:56.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:02:56.790 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:56.786+0000 7f825838c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:02:56.793 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:56.790+0000 7f825838c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:02:56.794 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:56.790+0000 7f825838c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:02:57.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:02:57.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:02:57.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:02:57.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:02:57.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:02:57.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:57.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:02:57.123 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:02:57.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:57.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:02:57.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:02:57.838 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:02:57.834+0000 7f825838c7c0 -1 Falling back to public interface 2024-09-13T15:02:58.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:02:58.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:58.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:02:58.466 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:02:58.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:58.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:02:58.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:02:59.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:02:59.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:02:59.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:02:59.786 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:02:59.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:02:59.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:03:00.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:03:00.567 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:00.566+0000 7f825838c7c0 -1 osd.0 94 log_to_monitors true 2024-09-13T15:03:01.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:03:01.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:03:01.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:03:01.109 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:03:01.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:03:01.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:03:01.508 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 98 up_thru 90 down_at 95 last_clean_interval [67,94) [v2:127.0.0.1:6802/3979606039,v1:127.0.0.1:6803/3979606039] [v2:127.0.0.1:6804/3979606039,v1:127.0.0.1:6805/3979606039] exists,up afd7d2c8-549f-44ac-ac39-8f66447cfc30 2024-09-13T15:03:01.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:03:01.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:03:01.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:03:01.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:03:01.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:03:01.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:03:01.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:03:01.509 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:03:01.510 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:03:01.510 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:03:01.510 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:03:01.510 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:03:01.510 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:03:01.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:03:01.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:03:01.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:03:01.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:03:01.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:03:01.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:03:01.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:03:01.994 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:03:01.994 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:03:01.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:03:01.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:03:01.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:03:02.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795010 2024-09-13T15:03:02.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795010 2024-09-13T15:03:02.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795010' 2024-09-13T15:03:02.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:03:02.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:03:02.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=386547056646 2024-09-13T15:03:02.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 386547056646 2024-09-13T15:03:02.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795010 1-386547056646' 2024-09-13T15:03:02.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:03:02.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:03:02.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350986 2024-09-13T15:03:02.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350986 2024-09-13T15:03:02.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795010 1-386547056646 2-347892350986' 2024-09-13T15:03:02.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:03:02.473 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-420906795010 2024-09-13T15:03:02.474 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:03:02.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:03:02.476 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-420906795010 2024-09-13T15:03:02.476 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:03:02.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795010 2024-09-13T15:03:02.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 420906795010' 2024-09-13T15:03:02.478 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 420906795010 2024-09-13T15:03:02.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:03:02.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808846 -lt 420906795010 2024-09-13T15:03:02.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:03:03.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:03:03.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:03:04.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795010 -lt 420906795010 2024-09-13T15:03:04.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:03:04.120 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-386547056646 2024-09-13T15:03:04.120 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:03:04.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:03:04.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-386547056646 2024-09-13T15:03:04.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:03:04.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=386547056646 2024-09-13T15:03:04.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 386547056646' 2024-09-13T15:03:04.125 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 386547056646 2024-09-13T15:03:04.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:03:04.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 386547056646 -lt 386547056646 2024-09-13T15:03:04.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:03:04.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-347892350986 2024-09-13T15:03:04.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:03:04.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:03:04.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-347892350986 2024-09-13T15:03:04.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:03:04.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350986 2024-09-13T15:03:04.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 347892350986' 2024-09-13T15:03:04.458 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 347892350986 2024-09-13T15:03:04.459 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:03:04.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350986 -lt 347892350986 2024-09-13T15:03:04.793 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:03:04.793 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:03:04.793 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:03:05.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:03:05.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:03:05.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:03:05.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:03:05.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:03:05.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:03:05.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:03:05.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:03:05.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-13T15:03:05.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:03:05.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:03:05.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:03:05.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-13T15:03:05.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-13T15:03:05.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:03:05.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-13T15:03:05.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:03:05.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:03:06.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:03:06.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:03:06.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:03:06.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:03:06.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:03:06.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:03:06.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:03:06.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:03:06.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-13T15:03:06.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:03:06.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:03:06.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:03:06.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-13T15:03:06.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-13T15:03:06.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:03:06.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:03:06.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:03:06.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:03:06.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:03:06.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:03:06.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:03:07.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:03:07.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:03:07.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T15:03:07.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:03:07.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T15:03:07.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:03:07.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:03:07.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:03:07.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:03:07.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:03:07.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:03:07.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:03:07.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:03:07.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:03:07.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:03:07.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:03:07.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:03:07.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:03:08.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:03:08.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:03:08.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-09-13T15:03:08.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-13T15:03:08.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:03:08.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:03:08.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:03:08.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:03:08.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-13T15:03:08.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:03:08.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:03:08.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:03:08.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:03:08.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:03:08.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:03:08.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:03:08.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:03:08.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:03:08.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:03:08.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:03:08.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-13T15:03:08.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:03:08.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:03:08.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:03:08.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:03:08.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:03:08.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-13T15:03:10.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:03:10.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:03:10.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:03:10.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:03:10.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:03:10.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:03:10.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:03:10.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:03:10.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:03:10.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:03:10.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:03:10.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:03:10.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:03:10.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:03:10.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:03:10.678 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:03:10.678 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:03:10.678 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:03:10.678 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:03:10.678 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:03:10.678 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:03:10.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:03:10.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:03:10.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:03:10.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:03:10.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:03:10.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:03:10.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:03:10.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:03:10.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:03:10.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:03:10.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:03:10.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:03:10.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:03:10.680 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:03:10.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:03:10.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:03:10.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:03:10.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:03:10.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:03:10.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:03:10.713 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:10.710+0000 7f3c0f4e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:03:10.716 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:10.714+0000 7f3c0f4e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:03:10.717 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:10.714+0000 7f3c0f4e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:03:11.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:03:11.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:03:11.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:03:11.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:03:11.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:03:11.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:03:11.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:03:11.014 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:03:11.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:03:11.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:03:11.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:03:12.022 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:12.018+0000 7f3c0f4e47c0 -1 Falling back to public interface 2024-09-13T15:03:12.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:03:12.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:03:12.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:03:12.345 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:03:12.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:03:12.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:03:12.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:03:13.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:03:13.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:03:13.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:03:13.684 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:03:13.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:03:13.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:03:14.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:03:14.720 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:14.718+0000 7f3c0f4e47c0 -1 osd.1 100 log_to_monitors true 2024-09-13T15:03:15.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:03:15.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:03:15.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:03:15.037 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:03:15.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:03:15.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:03:15.416 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 105 up_thru 98 down_at 101 last_clean_interval [90,100) [v2:127.0.0.1:6810/930086788,v1:127.0.0.1:6811/930086788] [v2:127.0.0.1:6812/930086788,v1:127.0.0.1:6813/930086788] exists,up 82f257eb-8a0f-4fa6-be3d-e20fb6b47202 2024-09-13T15:03:15.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:03:15.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:03:15.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:03:15.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:03:15.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:03:15.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:03:15.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:03:15.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:03:15.418 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:03:15.418 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:03:15.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:03:15.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:03:15.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:03:15.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:03:15.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:03:15.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:03:15.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:03:15.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:03:15.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:03:15.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:03:15.979 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:03:15.979 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:03:15.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:03:15.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:03:15.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:03:16.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795014 2024-09-13T15:03:16.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795014 2024-09-13T15:03:16.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795014' 2024-09-13T15:03:16.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:03:16.181 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:03:16.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566082 2024-09-13T15:03:16.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566082 2024-09-13T15:03:16.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795014 1-450971566082' 2024-09-13T15:03:16.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:03:16.375 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:03:16.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350990 2024-09-13T15:03:16.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350990 2024-09-13T15:03:16.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795014 1-450971566082 2-347892350990' 2024-09-13T15:03:16.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:03:16.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:03:16.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-420906795014 2024-09-13T15:03:16.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:03:16.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-420906795014 2024-09-13T15:03:16.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:03:16.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795014 2024-09-13T15:03:16.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 420906795014' 2024-09-13T15:03:16.540 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 420906795014 2024-09-13T15:03:16.540 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:03:16.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795012 -lt 420906795014 2024-09-13T15:03:16.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:03:17.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:03:17.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:03:18.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795014 -lt 420906795014 2024-09-13T15:03:18.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:03:18.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-450971566082 2024-09-13T15:03:18.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:03:18.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:03:18.223 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-450971566082 2024-09-13T15:03:18.223 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:03:18.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566082 2024-09-13T15:03:18.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 450971566082' 2024-09-13T15:03:18.224 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 450971566082 2024-09-13T15:03:18.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:03:18.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566082 -lt 450971566082 2024-09-13T15:03:18.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:03:18.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-347892350990 2024-09-13T15:03:18.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:03:18.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:03:18.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-347892350990 2024-09-13T15:03:18.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:03:18.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350990 2024-09-13T15:03:18.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 347892350990' 2024-09-13T15:03:18.537 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 347892350990 2024-09-13T15:03:18.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:03:18.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350990 -lt 347892350990 2024-09-13T15:03:18.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:03:18.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:03:18.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:03:19.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:03:19.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:03:19.275 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:03:19.276 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:03:19.276 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:03:19.276 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:03:19.276 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:03:19.276 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:03:19.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:03:19.603 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:03:19.603 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:03:19.603 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:03:19.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-13T15:03:19.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-13T15:03:19.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:03:19.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-13T15:03:19.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:03:19.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:03:20.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:03:20.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:03:20.088 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:03:20.089 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:03:20.089 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:03:20.089 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:03:20.089 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:03:20.089 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:03:20.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:03:20.419 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:03:20.419 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:03:20.419 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:03:20.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-13T15:03:20.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-13T15:03:20.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:03:20.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:03:20.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:03:20.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:03:20.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:03:20.788 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:03:20.788 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:03:21.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:03:21.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:03:21.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T15:03:21.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:03:21.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T15:03:21.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:03:21.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:03:21.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:03:21.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:03:21.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:03:21.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:03:21.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:03:21.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:03:21.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:03:21.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:03:21.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:03:21.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:03:22.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:03:22.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:03:22.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:03:22.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-13T15:03:22.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-09-13T15:03:22.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-09-13T15:03:22.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:03:22.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:03:22.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-09-13T15:03:22.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:03:22.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:03:22.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:03:22.570 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:03:22.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:03:23.004 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:03:23.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:03:23.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:03:23.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:03:23.286 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-09-13T15:03:23.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-13T15:03:23.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-13T15:03:23.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-13T15:03:23.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-13T15:03:23.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-09-13T15:03:23.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-13T15:03:23.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:03:23.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:03:23.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:03:23.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:03:23.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-13T15:03:23.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:03:23.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:03:23.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:03:23.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:03:23.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:03:23.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:03:23.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:03:23.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:03:23.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:03:23.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:03:23.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:03:23.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-13T15:03:23.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:03:23.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:03:23.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:03:23.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:03:23.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:03:23.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-09-13T15:03:26.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:03:26.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:03:26.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:03:26.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:03:26.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:03:26.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:03:26.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:03:26.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:03:26.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:03:26.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:03:26.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:03:26.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:03:26.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:03:26.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:03:26.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:03:26.174 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:03:26.174 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:03:26.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:03:26.176 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:03:26.176 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:03:26.176 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:03:26.176 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:03:26.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:03:26.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:03:26.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:03:26.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:03:26.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:03:26.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:03:26.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:03:26.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:03:26.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:03:26.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:03:26.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:03:26.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:03:26.177 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:03:26.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:03:26.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:03:26.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:03:26.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:03:26.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:03:26.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:03:26.211 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:26.205+0000 7f8c15eb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:03:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:26.209+0000 7f8c15eb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:03:26.215 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:26.213+0000 7f8c15eb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:03:26.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:03:26.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:03:26.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:03:26.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:03:26.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:03:26.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:03:26.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:03:26.514 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:03:26.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:03:26.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:03:26.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:03:27.482 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:27.477+0000 7f8c15eb97c0 -1 Falling back to public interface 2024-09-13T15:03:27.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:03:27.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:03:27.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:03:27.950 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:03:27.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:03:27.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:03:28.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:03:29.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:03:29.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:03:29.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:03:29.286 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:03:29.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:03:29.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:03:29.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:03:30.187 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:30.181+0000 7f8c15eb97c0 -1 osd.0 109 log_to_monitors true 2024-09-13T15:03:30.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:03:30.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:03:30.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:03:30.609 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:03:30.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:03:30.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:03:31.123 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 114 up_thru 105 down_at 110 last_clean_interval [98,109) [v2:127.0.0.1:6802/717324026,v1:127.0.0.1:6803/717324026] [v2:127.0.0.1:6804/717324026,v1:127.0.0.1:6805/717324026] exists,up afd7d2c8-549f-44ac-ac39-8f66447cfc30 2024-09-13T15:03:31.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:03:31.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:03:31.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:03:31.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:03:31.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:03:31.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:03:31.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:03:31.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:03:31.125 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:03:31.125 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:03:31.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:03:31.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:03:31.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:03:31.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:03:31.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:03:31.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:03:31.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:03:31.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:03:31.236 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:03:31.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:03:31.638 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:03:31.638 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:03:31.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:03:31.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:03:31.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:03:31.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=489626271746 2024-09-13T15:03:31.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 489626271746 2024-09-13T15:03:31.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-489626271746' 2024-09-13T15:03:31.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:03:31.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:03:31.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566086 2024-09-13T15:03:31.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566086 2024-09-13T15:03:31.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-489626271746 1-450971566086' 2024-09-13T15:03:31.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:03:31.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:03:32.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350994 2024-09-13T15:03:32.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350994 2024-09-13T15:03:32.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-489626271746 1-450971566086 2-347892350994' 2024-09-13T15:03:32.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:03:32.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-489626271746 2024-09-13T15:03:32.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:03:32.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:03:32.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-489626271746 2024-09-13T15:03:32.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:03:32.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=489626271746 2024-09-13T15:03:32.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 489626271746' 2024-09-13T15:03:32.116 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 489626271746 2024-09-13T15:03:32.117 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:03:32.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 489626271746 2024-09-13T15:03:32.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:03:33.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:03:33.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:03:33.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 489626271746 -lt 489626271746 2024-09-13T15:03:33.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:03:33.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-450971566086 2024-09-13T15:03:33.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:03:33.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:03:33.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-450971566086 2024-09-13T15:03:33.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:03:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566086 2024-09-13T15:03:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 450971566086' 2024-09-13T15:03:33.766 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 450971566086 2024-09-13T15:03:33.766 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:03:34.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566086 -lt 450971566086 2024-09-13T15:03:34.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:03:34.084 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-347892350994 2024-09-13T15:03:34.084 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:03:34.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:03:34.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-347892350994 2024-09-13T15:03:34.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:03:34.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350994 2024-09-13T15:03:34.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 347892350994' 2024-09-13T15:03:34.088 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 347892350994 2024-09-13T15:03:34.089 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:03:34.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350994 -lt 347892350994 2024-09-13T15:03:34.419 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:03:34.419 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:03:34.420 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:03:34.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:03:34.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:03:34.810 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:03:34.810 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:03:34.810 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:03:34.810 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:03:34.810 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:03:34.810 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:03:35.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:03:35.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:03:35.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:03:35.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:03:35.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:03:35.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:03:35.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:03:35.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-13T15:03:35.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:03:35.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:03:35.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:03:35.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:03:35.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-13T15:03:35.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:03:35.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:03:35.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:03:35.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:03:35.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:03:35.536 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:03:35.536 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:03:35.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:03:35.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:03:35.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:03:35.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:03:35.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-13T15:03:35.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:03:35.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:03:35.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:03:35.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:03:35.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:03:35.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-13T15:03:38.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:03:38.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:03:38.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:03:38.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:03:38.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:03:38.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:03:38.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:03:38.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:03:38.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:03:38.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:03:38.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:03:38.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:03:38.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:03:38.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:03:38.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:03:38.581 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:03:38.581 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:03:38.581 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:03:38.581 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:03:38.581 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:03:38.581 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:03:38.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:03:38.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:03:38.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:03:38.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:03:38.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:03:38.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:03:38.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:03:38.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:03:38.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:03:38.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:03:38.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:03:38.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:03:38.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:03:38.583 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:03:38.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:03:38.584 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:03:38.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:03:38.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:03:38.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:03:38.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:03:38.625 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:38.621+0000 7fe317b707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:03:38.628 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:38.625+0000 7fe317b707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:03:38.629 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:38.625+0000 7fe317b707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:03:38.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:03:38.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:03:38.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:03:38.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:03:38.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:03:38.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:03:38.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:03:38.929 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:03:38.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:03:38.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:03:39.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:03:39.686 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:39.681+0000 7fe317b707c0 -1 Falling back to public interface 2024-09-13T15:03:40.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:03:40.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:03:40.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:03:40.280 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:03:40.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:03:40.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:03:40.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:03:41.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:03:41.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:03:41.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:03:41.620 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:03:41.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:03:41.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:03:41.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:03:42.399 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:03:42.393+0000 7fe317b707c0 -1 osd.0 115 log_to_monitors true 2024-09-13T15:03:42.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:03:42.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:03:42.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:03:42.958 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:03:42.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:03:42.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:03:43.344 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 120 up_thru 120 down_at 116 last_clean_interval [114,115) [v2:127.0.0.1:6802/2203059765,v1:127.0.0.1:6803/2203059765] [v2:127.0.0.1:6804/2203059765,v1:127.0.0.1:6805/2203059765] exists,up afd7d2c8-549f-44ac-ac39-8f66447cfc30 2024-09-13T15:03:43.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:03:43.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:03:43.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:03:43.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:03:43.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:03:43.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:03:43.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:03:43.345 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:03:43.346 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:03:43.346 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:03:43.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:03:43.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:03:43.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:03:43.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:03:43.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:03:43.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:03:43.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:03:43.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:03:43.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:03:43.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:03:43.818 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:03:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:03:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:03:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:03:43.819 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:03:44.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=515396075522 2024-09-13T15:03:44.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 515396075522 2024-09-13T15:03:44.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-515396075522' 2024-09-13T15:03:44.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:03:44.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:03:44.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566089 2024-09-13T15:03:44.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566089 2024-09-13T15:03:44.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-515396075522 1-450971566089' 2024-09-13T15:03:44.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:03:44.165 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:03:44.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350997 2024-09-13T15:03:44.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350997 2024-09-13T15:03:44.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-515396075522 1-450971566089 2-347892350997' 2024-09-13T15:03:44.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:03:44.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-515396075522 2024-09-13T15:03:44.311 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:03:44.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:03:44.312 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-515396075522 2024-09-13T15:03:44.312 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:03:44.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=515396075522 2024-09-13T15:03:44.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 515396075522' 2024-09-13T15:03:44.313 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 515396075522 2024-09-13T15:03:44.314 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:03:44.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 515396075522 2024-09-13T15:03:44.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:03:45.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:03:45.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:03:45.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 515396075522 -lt 515396075522 2024-09-13T15:03:45.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:03:45.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-450971566089 2024-09-13T15:03:45.954 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:03:45.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:03:45.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-450971566089 2024-09-13T15:03:45.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:03:45.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566089 2024-09-13T15:03:45.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 450971566089' 2024-09-13T15:03:45.958 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 450971566089 2024-09-13T15:03:45.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:03:46.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566089 -lt 450971566089 2024-09-13T15:03:46.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:03:46.294 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-347892350997 2024-09-13T15:03:46.294 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:03:46.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:03:46.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-347892350997 2024-09-13T15:03:46.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:03:46.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350997 2024-09-13T15:03:46.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 347892350997' 2024-09-13T15:03:46.299 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 347892350997 2024-09-13T15:03:46.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:03:46.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350997 -lt 347892350997 2024-09-13T15:03:46.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:03:46.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:03:46.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:03:47.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:03:47.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:03:47.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:03:47.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:03:47.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:03:47.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:03:47.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:03:47.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:03:47.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:03:47.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:03:47.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:03:47.370 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:03:47.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:03:47.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:03:47.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:03:47.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-09-13T15:03:47.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-09-13T15:03:47.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T15:03:47.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=EOBJ0 2024-09-13T15:03:47.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool EOBJ0 2024-09-13T15:03:47.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T15:03:48.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-09-13T15:03:48.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-09-13T15:03:48.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=3.0 2024-09-13T15:03:48.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 3.0 2024-09-13T15:03:48.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:03:48.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:03:48.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:03:48.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:03:48.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-13T15:01:01.317277+0000 2024-09-13T15:03:48.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 3.0 2024-09-13T15:03:48.672 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-09-13T15:03:48.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 3.0 2024-09-13T15:01:01.317277+0000 2024-09-13T15:03:48.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-13T15:03:48.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:01:01.317277+0000 2024-09-13T15:03:48.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:03:48.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:03:48.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:03:48.689 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-13T15:03:48.689 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:03:48.689 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:03:48.690 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:03:48.690 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:03:49.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:01:01.317277+0000 '>' 2024-09-13T15:01:01.317277+0000 2024-09-13T15:03:49.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:03:50.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:03:50.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:03:50.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-13T15:03:50.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:03:50.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:03:50.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:03:50.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:03:50.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:01:01.317277+0000 '>' 2024-09-13T15:01:01.317277+0000 2024-09-13T15:03:50.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:03:51.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:03:51.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:03:51.339 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-13T15:03:51.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:03:51.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:03:51.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:03:51.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:03:51.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:03:48.966599+0000 '>' 2024-09-13T15:01:01.317277+0000 2024-09-13T15:03:51.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:03:51.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-13T15:03:51.690 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-13T15:03:51.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-09-13T15:03:51.717 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-13T15:03:51.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-13T15:03:51.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-13T15:03:51.769 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-13T15:03:51.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=120 2024-09-13T15:03:51.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-13T15:03:51.795 INFO:tasks.workunit.client.0.smithi112.stderr: . as $in 2024-09-13T15:03:51.795 INFO:tasks.workunit.client.0.smithi112.stderr: | if type == "object" then 2024-09-13T15:03:51.795 INFO:tasks.workunit.client.0.smithi112.stderr: reduce keys[] as $key 2024-09-13T15:03:51.795 INFO:tasks.workunit.client.0.smithi112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-13T15:03:51.795 INFO:tasks.workunit.client.0.smithi112.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-13T15:03:51.795 INFO:tasks.workunit.client.0.smithi112.stderr: else f 2024-09-13T15:03:51.795 INFO:tasks.workunit.client.0.smithi112.stderr: end; 2024-09-13T15:03:51.795 INFO:tasks.workunit.client.0.smithi112.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-13T15:03:51.795 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-13T15:03:51.795 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-13T15:03:51.795 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-13T15:03:51.795 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-13T15:03:51.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-09-13T15:03:51.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-13T15:03:51.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-13T15:03:51.834 INFO:tasks.workunit.client.0.smithi112.stderr: . as $in 2024-09-13T15:03:51.834 INFO:tasks.workunit.client.0.smithi112.stderr: | if type == "object" then 2024-09-13T15:03:51.835 INFO:tasks.workunit.client.0.smithi112.stderr: reduce keys[] as $key 2024-09-13T15:03:51.835 INFO:tasks.workunit.client.0.smithi112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-13T15:03:51.835 INFO:tasks.workunit.client.0.smithi112.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-13T15:03:51.835 INFO:tasks.workunit.client.0.smithi112.stderr: else f 2024-09-13T15:03:51.835 INFO:tasks.workunit.client.0.smithi112.stderr: end; 2024-09-13T15:03:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-13T15:03:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-13T15:03:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-13T15:03:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-13T15:03:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-13T15:03:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-09-13T15:03:51.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-13T15:03:51.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-13T15:03:51.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-13T15:03:51.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-09-13T15:03:51.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-09-13T15:03:51.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-09-13T15:03:51.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-13T15:03:51.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:03:51.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:03:51.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:03:51.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:03:51.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:03:52.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-13T15:01:01.317277+0000 2024-09-13T15:03:52.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-13T15:03:52.502 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-09-13T15:03:52.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-13T15:01:01.317277+0000 last_deep_scrub_stamp 2024-09-13T15:03:52.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-13T15:03:52.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:01:01.317277+0000 2024-09-13T15:03:52.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-13T15:03:52.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:03:52.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:03:52.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:03:52.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:03:52.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:03:52.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:03:52.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:03:52.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:01:01.317277+0000 '>' 2024-09-13T15:01:01.317277+0000 2024-09-13T15:03:52.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:03:53.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:03:53.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:03:53.822 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:03:53.822 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:03:53.822 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:03:53.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:03:53.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:03:54.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:01:01.317277+0000 '>' 2024-09-13T15:01:01.317277+0000 2024-09-13T15:03:54.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:03:55.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:03:55.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:03:55.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:03:55.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:03:55.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:03:55.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:03:55.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:03:55.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:01:01.317277+0000 '>' 2024-09-13T15:01:01.317277+0000 2024-09-13T15:03:55.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:03:56.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:03:56.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:03:56.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:03:56.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:03:56.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:03:56.482 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:03:56.482 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:03:56.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:01:01.317277+0000 '>' 2024-09-13T15:01:01.317277+0000 2024-09-13T15:03:56.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:03:57.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:03:57.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:03:57.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:03:57.819 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:03:57.819 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:03:57.819 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:03:57.819 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:03:58.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:03:53.078588+0000 '>' 2024-09-13T15:01:01.317277+0000 2024-09-13T15:03:58.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:03:58.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-13T15:03:58.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-13T15:03:58.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-09-13T15:03:58.205 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-13T15:03:58.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-13T15:03:58.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-13T15:03:58.269 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-13T15:03:58.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=120 2024-09-13T15:03:58.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-09-13T15:03:58.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-13T15:03:58.298 INFO:tasks.workunit.client.0.smithi112.stderr: . as $in 2024-09-13T15:03:58.298 INFO:tasks.workunit.client.0.smithi112.stderr: | if type == "object" then 2024-09-13T15:03:58.298 INFO:tasks.workunit.client.0.smithi112.stderr: reduce keys[] as $key 2024-09-13T15:03:58.298 INFO:tasks.workunit.client.0.smithi112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-13T15:03:58.299 INFO:tasks.workunit.client.0.smithi112.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-13T15:03:58.299 INFO:tasks.workunit.client.0.smithi112.stderr: else f 2024-09-13T15:03:58.299 INFO:tasks.workunit.client.0.smithi112.stderr: end; 2024-09-13T15:03:58.299 INFO:tasks.workunit.client.0.smithi112.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-13T15:03:58.299 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-13T15:03:58.299 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-13T15:03:58.299 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-13T15:03:58.299 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-13T15:03:58.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-09-13T15:03:58.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-13T15:03:58.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-13T15:03:58.343 INFO:tasks.workunit.client.0.smithi112.stderr: . as $in 2024-09-13T15:03:58.343 INFO:tasks.workunit.client.0.smithi112.stderr: | if type == "object" then 2024-09-13T15:03:58.343 INFO:tasks.workunit.client.0.smithi112.stderr: reduce keys[] as $key 2024-09-13T15:03:58.343 INFO:tasks.workunit.client.0.smithi112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-13T15:03:58.343 INFO:tasks.workunit.client.0.smithi112.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-13T15:03:58.344 INFO:tasks.workunit.client.0.smithi112.stderr: else f 2024-09-13T15:03:58.344 INFO:tasks.workunit.client.0.smithi112.stderr: end; 2024-09-13T15:03:58.344 INFO:tasks.workunit.client.0.smithi112.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-13T15:03:58.344 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-13T15:03:58.344 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-13T15:03:58.344 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-13T15:03:58.345 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-13T15:03:58.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-09-13T15:03:58.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-13T15:03:58.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-13T15:03:58.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-13T15:03:58.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-09-13T15:03:58.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-09-13T15:03:58.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-09-13T15:03:58.778 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'ecpool' removed 2024-09-13T15:03:58.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T15:03:58.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:03:58.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:03:58.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:03:58.796 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:03:58.796 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:03:58.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:03:58.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:03:58.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:03:59.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:03:59.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:03:59.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:03:59.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:03:59.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:03:59.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:03:59.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:03:59.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:03:59.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:03:59.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:03:59.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:03:59.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:03:59.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:03:59.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:03:59.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:03:59.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:03:59.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:03:59.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:03:59.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:03:59.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:03:59.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T15:03:59.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T15:03:59.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T15:03:59.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T15:03:59.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:03:59.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:03:59.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:03:59.156 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:03:59.156 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:03:59.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:03:59.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:03:59.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:03:59.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:03:59.159 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:03:59.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:03:59.161 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:03:59.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:03:59.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:03:59.162 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:03:59.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:03:59.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:03:59.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:03:59.164 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:03:59.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:03:59.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:03:59.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:03:59.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:03:59.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:03:59.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:03:59.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:03:59.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:03:59.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:03:59.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T15:03:59.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:03:59.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:03:59.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:03:59.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T15:03:59.174 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:03:59.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T15:03:59.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:03:59.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T15:03:59.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-09-13T15:03:59.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-09-13T15:03:59.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-09-13T15:03:59.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-09-13T15:03:59.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-09-13T15:03:59.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-09-13T15:03:59.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-09-13T15:03:59.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-09-13T15:03:59.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T15:03:59.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:03:59.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:03:59.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:03:59.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T15:03:59.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T15:03:59.647 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:03:59.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:03:59.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:03:59.649 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:03:59.649 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:03:59.649 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:03:59.649 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:03:59.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T15:03:59.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:03:59.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:03:59.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:03:59.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:03:59.869 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:03:59.869 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:03:59.869 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:03:59.869 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:03:59.869 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:03:59.870 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:03:59.870 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:03:59.870 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:03:59.870 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:03:59.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:03:59.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T15:03:59.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:03:59.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:03:59.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:03:59.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:03:59.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:03:59.983 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:03:59.983 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:03:59.983 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:03:59.984 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:03:59.984 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:03:59.984 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:03:59.984 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:03:59.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:03:59.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T15:04:00.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-09-13T15:04:00.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T15:04:00.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:04:00.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:04:00.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:04:00.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T15:04:00.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:04:00.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:04:00.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:04:00.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:04:00.958 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:04:00.958 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:04:00.959 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:04:00.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:04:00.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:04:00.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:04:00.996 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-09-13T15:04:01.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-13T15:04:01.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-09-13T15:04:01.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:04:01.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:04:01.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:04:01.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:04:01.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:04:01.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:04:01.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:04:01.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:04:01.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:04:01.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:04:01.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:04:01.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:04:01.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:04:01.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:04:01.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:04:01.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:04:01.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:04:01.001 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:04:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:04:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:04:01.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:04:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:04:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:04:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:04:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:04:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:04:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:04:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:04:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:04:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:04:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:04:01.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:04:01.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:04:01.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:04:01.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:04:01.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=51e95a12-0e9f-4e04-84e7-690a88ab0006 2024-09-13T15:04:01.004 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 51e95a12-0e9f-4e04-84e7-690a88ab0006 2024-09-13T15:04:01.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 51e95a12-0e9f-4e04-84e7-690a88ab0006' 2024-09-13T15:04:01.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:04:01.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBhVORm4yMtARAAmTOE1dtTj3scYb+rItp8PQ== 2024-09-13T15:04:01.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBhVORm4yMtARAAmTOE1dtTj3scYb+rItp8PQ=="}' 2024-09-13T15:04:01.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 51e95a12-0e9f-4e04-84e7-690a88ab0006 -i td/osd-scrub-repair/0/new.json 2024-09-13T15:04:01.253 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:04:01.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T15:04:01.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhVORm4yMtARAAmTOE1dtTj3scYb+rItp8PQ== --osd-uuid 51e95a12-0e9f-4e04-84e7-690a88ab0006 2024-09-13T15:04:01.305 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:01.300+0000 7faf6db3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:04:01.307 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:01.300+0000 7faf6db3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:04:01.308 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:01.304+0000 7faf6db3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:04:01.309 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:01.304+0000 7faf6db3f7c0 -1 bdev(0x5566c5fefc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:04:01.309 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:01.304+0000 7faf6db3f7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T15:04:07.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T15:04:07.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:04:07.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:04:07.473 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:04:07.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:04:07.882 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:04:07.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:04:07.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:04:07.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:04:07.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:04:07.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:04:07.908 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:07.904+0000 7fed380e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:04:07.910 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:07.904+0000 7fed380e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:04:07.912 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:07.908+0000 7fed380e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:04:08.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:04:08.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:04:08.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:04:08.206 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:04:08.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:04:08.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:04:08.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:08.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:04:08.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:08.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:08.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:09.218 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:09.216+0000 7fed380e67c0 -1 Falling back to public interface 2024-09-13T15:04:09.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:09.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:09.550 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:04:09.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:04:09.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:09.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:09.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:10.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:10.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:10.873 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:04:10.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:04:10.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:10.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:11.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:11.871 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:11.868+0000 7fed380e67c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:04:12.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:12.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:12.202 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:04:12.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:04:12.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:12.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:12.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:13.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:13.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:13.532 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:04:13.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:04:13.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:13.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:13.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:14.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:14.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:14.876 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:04:14.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:04:14.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:14.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:15.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:16.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:16.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:16.210 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:04:16.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:04:16.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:16.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:16.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:17.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:17.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:17.611 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:04:17.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:04:17.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:17.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:17.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:18.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:18.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:18.946 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:04:18.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:04:18.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:18.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:19.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:20.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:20.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:20.279 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:04:20.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:04:20.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:20.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:20.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:21.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:21.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:21.652 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:04:21.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:04:21.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:21.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:21.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:22.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:22.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:22.972 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:04:22.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:04:22.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:22.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:23.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:24.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:24.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:24.291 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:04:24.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:04:24.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:24.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:25.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:26.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:26.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:26.039 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:04:26.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:04:26.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:26.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:26.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:27.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:27.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:27.553 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:04:27.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:04:27.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:27.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:28.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:29.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:29.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:29.220 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:04:29.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:04:29.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:29.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:29.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:30.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:30.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:30.778 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:04:30.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:04:30.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:30.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:31.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:32.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:32.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:32.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:04:32.378 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:04:32.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:32.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:33.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:34.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:34.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:34.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:04:34.136 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:04:34.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:34.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:04:34.530 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4091759792,v1:127.0.0.1:6803/4091759792] [v2:127.0.0.1:6804/4091759792,v1:127.0.0.1:6805/4091759792] exists,up 51e95a12-0e9f-4e04-84e7-690a88ab0006 2024-09-13T15:04:34.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:04:34.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:04:34.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:04:34.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-13T15:04:34.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-09-13T15:04:34.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:04:34.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:04:34.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T15:04:34.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:04:34.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:04:34.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:04:34.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:04:34.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:04:34.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:04:34.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:04:34.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:04:34.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:04:34.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:04:34.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:04:34.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:04:34.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:04:34.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:04:34.535 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:04:34.535 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:04:34.535 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:04:34.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:04:34.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:04:34.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:04:34.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:04:34.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:04:34.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:04:34.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:04:34.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:04:34.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:04:34.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:04:34.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:04:34.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:04:34.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:04:34.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:04:34.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:04:34.538 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 7c0aa1ce-2183-4848-8274-f45e71e8ba57 2024-09-13T15:04:34.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7c0aa1ce-2183-4848-8274-f45e71e8ba57 2024-09-13T15:04:34.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7c0aa1ce-2183-4848-8274-f45e71e8ba57' 2024-09-13T15:04:34.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:04:34.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCCVORmLesvIRAAYhj+YIy+PeZy00Hm5k1U5Q== 2024-09-13T15:04:34.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCCVORmLesvIRAAYhj+YIy+PeZy00Hm5k1U5Q=="}' 2024-09-13T15:04:34.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7c0aa1ce-2183-4848-8274-f45e71e8ba57 -i td/osd-scrub-repair/1/new.json 2024-09-13T15:04:34.922 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:04:34.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T15:04:34.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCCVORmLesvIRAAYhj+YIy+PeZy00Hm5k1U5Q== --osd-uuid 7c0aa1ce-2183-4848-8274-f45e71e8ba57 2024-09-13T15:04:34.974 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:34.971+0000 7f7ac2fd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:04:35.016 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:35.011+0000 7f7ac2fd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:04:35.017 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:35.015+0000 7f7ac2fd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:04:35.017 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:35.015+0000 7f7ac2fd77c0 -1 bdev(0x560f40505c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T15:04:35.017 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:35.015+0000 7f7ac2fd77c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T15:04:41.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T15:04:41.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:04:41.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T15:04:41.427 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T15:04:41.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:04:41.841 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:04:41.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T15:04:41.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:04:41.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:04:41.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:04:41.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:04:41.874 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:41.871+0000 7f1be5f1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:04:41.886 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:41.883+0000 7f1be5f1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:04:41.888 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:41.887+0000 7f1be5f1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:04:42.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T15:04:42.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:04:42.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:04:42.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:04:42.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:04:42.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:42.160 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:04:42.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:04:42.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:42.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:04:42.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:43.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:43.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:43.467 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:04:43.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:04:43.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:43.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:04:43.481 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:43.479+0000 7f1be5f1d7c0 -1 Falling back to public interface 2024-09-13T15:04:43.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:44.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:44.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:44.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:04:44.809 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:04:44.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:44.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:04:45.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:46.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:46.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:46.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:04:46.134 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:04:46.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:46.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:04:46.289 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:04:46.287+0000 7f1be5f1d7c0 -1 osd.1 0 log_to_monitors true 2024-09-13T15:04:46.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:47.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:47.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:47.498 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:04:47.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:04:47.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:47.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:04:47.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:48.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:48.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:48.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:04:48.842 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:04:48.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:48.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:04:49.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:50.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:50.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:50.327 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:04:50.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:04:50.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:50.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:04:50.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:51.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:51.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:51.657 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:04:51.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:04:51.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:51.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:04:52.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:53.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:53.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:53.009 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:04:53.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:04:53.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:53.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:04:53.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:54.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:54.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:54.342 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:04:54.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:04:54.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:54.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:04:54.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:55.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:55.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:55.732 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:04:55.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:04:55.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:55.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:04:56.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:57.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:57.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:57.145 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:04:57.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:04:57.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:57.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:04:57.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:58.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:58.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:58.476 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:04:58.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:04:58.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:58.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:04:58.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:04:59.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:04:59.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:04:59.814 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:04:59.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:04:59.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:04:59.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:05:00.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:01.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:01.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:01.717 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:05:01.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:05:01.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:01.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:05:02.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:03.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:03.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:03.737 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:05:03.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:05:03.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:03.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:05:04.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:05.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:05.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:05.264 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:05:05.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:05:05.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:05.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:05:05.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:06.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:06.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:06.835 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:05:06.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:05:06.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:06.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:05:07.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:08.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:08.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:08.584 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:05:08.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:05:08.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:08.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:05:09.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:10.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:10.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:10.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:05:10.012 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:05:10.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:10.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:05:10.456 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1209156862,v1:127.0.0.1:6811/1209156862] [v2:127.0.0.1:6812/1209156862,v1:127.0.0.1:6813/1209156862] exists,up 7c0aa1ce-2183-4848-8274-f45e71e8ba57 2024-09-13T15:05:10.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:05:10.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:05:10.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:05:10.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-13T15:05:10.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-09-13T15:05:10.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:05:10.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:05:10.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T15:05:10.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:05:10.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:05:10.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:05:10.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:05:10.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:05:10.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:05:10.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T15:05:10.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T15:05:10.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:05:10.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:05:10.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:05:10.462 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:05:10.462 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:05:10.462 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:05:10.462 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:05:10.462 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:05:10.462 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:05:10.462 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:05:10.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:05:10.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:05:10.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:05:10.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:05:10.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:05:10.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:05:10.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:05:10.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:05:10.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:05:10.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:05:10.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:05:10.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:05:10.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T15:05:10.464 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:05:10.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=53e74c87-c90b-44c8-99df-7b44aa117487 2024-09-13T15:05:10.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 53e74c87-c90b-44c8-99df-7b44aa117487' 2024-09-13T15:05:10.465 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 53e74c87-c90b-44c8-99df-7b44aa117487 2024-09-13T15:05:10.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:05:10.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCmVORmHyW+HBAAWbOHjg9ObEKJhYoWj+ekzg== 2024-09-13T15:05:10.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCmVORmHyW+HBAAWbOHjg9ObEKJhYoWj+ekzg=="}' 2024-09-13T15:05:10.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 53e74c87-c90b-44c8-99df-7b44aa117487 -i td/osd-scrub-repair/2/new.json 2024-09-13T15:05:10.864 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:05:10.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T15:05:10.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCmVORmHyW+HBAAWbOHjg9ObEKJhYoWj+ekzg== --osd-uuid 53e74c87-c90b-44c8-99df-7b44aa117487 2024-09-13T15:05:10.919 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:05:10.914+0000 7fa6593ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:05:10.940 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:05:10.938+0000 7fa6593ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:05:10.941 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:05:10.938+0000 7fa6593ef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:05:10.942 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:05:10.938+0000 7fa6593ef7c0 -1 bdev(0x55d6f6ab3c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T15:05:10.942 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:05:10.938+0000 7fa6593ef7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T15:05:17.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T15:05:17.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:05:17.432 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T15:05:17.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T15:05:17.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:05:17.824 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T15:05:17.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T15:05:17.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:05:17.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:05:17.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:05:17.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:05:17.858 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:05:17.854+0000 7fc1bb25a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:05:17.861 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:05:17.858+0000 7fc1bb25a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:05:17.862 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:05:17.858+0000 7fc1bb25a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:05:18.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T15:05:18.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:05:18.174 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:05:18.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:05:18.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:05:18.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:05:18.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:18.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:05:18.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:18.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:18.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:18.942 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:05:18.938+0000 7fc1bb25a7c0 -1 Falling back to public interface 2024-09-13T15:05:19.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:19.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:19.501 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:05:19.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:05:19.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:19.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:19.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:20.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:20.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:20.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:05:20.835 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:05:20.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:20.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:21.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:21.750 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:05:21.746+0000 7fc1bb25a7c0 -1 osd.2 0 log_to_monitors true 2024-09-13T15:05:22.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:22.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:22.160 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:05:22.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:05:22.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:22.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:22.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:23.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:23.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:23.506 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:05:23.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:05:23.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:23.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:23.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:24.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:24.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:24.844 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:05:24.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:05:24.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:24.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:25.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:26.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:26.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:26.209 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:05:26.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:05:26.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:26.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:26.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:27.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:27.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:27.684 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:05:27.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:05:27.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:27.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:28.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:29.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:29.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:29.007 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:05:29.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:05:29.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:29.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:29.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:30.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:30.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:30.386 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:05:30.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:05:30.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:30.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:30.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:31.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:31.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:31.849 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:05:31.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:05:31.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:31.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:32.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:33.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:33.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:33.179 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:05:33.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:05:33.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:33.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:33.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:34.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:34.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:34.599 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:05:34.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:05:34.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:34.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:35.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:36.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:36.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:36.039 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:05:36.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:05:36.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:36.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:36.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:37.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:37.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:37.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:05:37.508 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:05:37.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:37.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:38.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:39.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:39.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:39.138 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:05:39.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:05:39.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:39.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:39.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:40.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:40.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:40.520 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:05:40.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:05:40.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:40.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:41.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:42.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:42.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:42.183 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:05:42.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:05:42.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:42.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:42.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:43.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:43.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:43.588 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:05:43.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:05:43.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:43.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:44.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:05:45.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:05:45.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:05:45.409 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:05:45.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:05:45.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:05:45.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:05:45.757 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1354510235,v1:127.0.0.1:6819/1354510235] [v2:127.0.0.1:6820/1354510235,v1:127.0.0.1:6821/1354510235] exists,up 53e74c87-c90b-44c8-99df-7b44aa117487 2024-09-13T15:05:45.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:05:45.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:05:45.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:05:45.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-09-13T15:05:45.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T15:05:46.082 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' does not exist 2024-09-13T15:05:46.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T15:05:46.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T15:05:46.524 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T15:05:46.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:05:47.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T15:05:49.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-09-13T15:05:49.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2024-09-13T15:05:49.943 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'foo' created 2024-09-13T15:05:49.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:05:50.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-09-13T15:05:50.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-13T15:05:50.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-13T15:05:50.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-13T15:05:50.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-13T15:05:50.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-09-13T15:05:51.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-13T15:05:51.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-13T15:05:52.421 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'ecpool' created 2024-09-13T15:05:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:05:53.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-13T15:05:53.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-13T15:05:53.985 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 3 allow_ec_overwrites to true 2024-09-13T15:05:54.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-13T15:05:54.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:05:54.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:05:54.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:05:54.008 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:05:54.008 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:05:54.008 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:05:54.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:05:54.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:05:54.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:05:54.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:05:54.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:05:54.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:05:54.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:05:54.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:05:54.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:05:54.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:05:54.464 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:05:54.464 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:05:54.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:05:54.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:05:54.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:05:54.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836500 2024-09-13T15:05:54.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836500 2024-09-13T15:05:54.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500' 2024-09-13T15:05:54.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:05:54.666 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:05:54.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672975 2024-09-13T15:05:54.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672975 2024-09-13T15:05:54.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672975' 2024-09-13T15:05:54.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:05:54.810 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:05:54.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509446 2024-09-13T15:05:54.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509446 2024-09-13T15:05:54.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672975 2-64424509446' 2024-09-13T15:05:54.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:05:54.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836500 2024-09-13T15:05:54.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:05:54.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:05:54.955 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836500 2024-09-13T15:05:54.955 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:05:54.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836500 2024-09-13T15:05:54.956 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836500 2024-09-13T15:05:54.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-13T15:05:54.956 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:05:55.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836500 2024-09-13T15:05:55.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:05:56.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:05:56.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:05:56.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836500 2024-09-13T15:05:56.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:05:56.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672975 2024-09-13T15:05:56.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:05:56.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:05:56.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672975 2024-09-13T15:05:56.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:05:56.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672975 2024-09-13T15:05:56.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-13T15:05:56.597 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672975 2024-09-13T15:05:56.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:05:57.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-13T15:05:57.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:05:57.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509446 2024-09-13T15:05:57.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:05:57.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:05:57.007 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509446 2024-09-13T15:05:57.007 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:05:57.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509446 2024-09-13T15:05:57.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-13T15:05:57.009 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509446 2024-09-13T15:05:57.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:05:57.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509446 2024-09-13T15:05:57.364 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:05:57.365 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:05:57.365 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:05:57.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:05:57.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:05:57.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:05:57.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:05:57.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:05:57.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:05:57.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:05:57.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:05:58.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:05:58.101 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:05:58.101 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:05:58.101 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:05:58.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:05:58.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:05:58.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:05:58.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-13T15:05:58.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-09-13T15:05:58.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:05:58.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:05:58.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:05:58.470 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:05:58.470 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:05:58.470 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:05:58.470 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:05:58.470 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:05:58.470 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:05:58.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:05:58.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:05:58.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:05:58.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:05:58.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:05:58.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:05:58.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:05:58.911 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:05:58.911 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:05:58.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:05:58.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:05:58.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:05:59.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-13T15:05:59.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-13T15:05:59.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-13T15:05:59.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:05:59.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:05:59.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-13T15:05:59.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-13T15:05:59.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672977' 2024-09-13T15:05:59.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:05:59.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:05:59.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-13T15:05:59.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-13T15:05:59.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672977 2-64424509448' 2024-09-13T15:05:59.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:05:59.379 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-13T15:05:59.379 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:05:59.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:05:59.382 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:05:59.382 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-13T15:05:59.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-13T15:05:59.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-13T15:05:59.383 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836502 2024-09-13T15:05:59.383 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:05:59.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836502 2024-09-13T15:05:59.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:06:00.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:06:00.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:06:01.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-13T15:06:01.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:06:01.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-13T15:06:01.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:06:01.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:06:01.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:06:01.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-13T15:06:01.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-13T15:06:01.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-13T15:06:01.020 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672977 2024-09-13T15:06:01.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:06:01.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-13T15:06:01.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:06:01.372 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-13T15:06:01.372 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:06:01.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:06:01.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-13T15:06:01.375 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:06:01.376 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509448 2024-09-13T15:06:01.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-13T15:06:01.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-13T15:06:01.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:06:01.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-13T15:06:01.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:06:01.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:06:01.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:06:02.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:06:02.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:06:02.110 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:06:02.110 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:06:02.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:06:02.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:06:02.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:06:02.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:06:02.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:06:02.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:06:02.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:06:02.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:06:02.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:06:02.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:06:02.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:06:02.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-09-13T15:06:02.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-13T15:06:02.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-09-13T15:06:02.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-09-13T15:06:02.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:06:02.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:06:02.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-09-13T15:06:02.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:06:02.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:06:02.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:06:03.380 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:06:03.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:06:03.916 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:06:03.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:06:03.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:06:03.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:06:04.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-09-13T15:06:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-13T15:06:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-13T15:06:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-09-13T15:06:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-09-13T15:06:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:06:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:06:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:06:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:06:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:06:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:06:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:06:04.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:06:04.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:06:04.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:06:04.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:06:04.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:06:04.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:06:04.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:06:04.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:06:04.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:06:05.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:06:05.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:06:05.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:06:05.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:06:05.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:06:05.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:06:05.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:06:05.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:06:07.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:06:07.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:06:07.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:06:07.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:06:07.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:06:07.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:06:07.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:06:07.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:06:07.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:06:07.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:06:07.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:06:07.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:06:07.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:06:07.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:06:07.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:06:07.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:06:07.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:06:07.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:06:07.612 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:06:07.612 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:06:07.612 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:06:07.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:06:07.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:06:07.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:06:07.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:06:07.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:06:07.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:06:07.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:06:07.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:06:07.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:06:07.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:06:07.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:06:07.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:06:07.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:06:07.615 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:06:07.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:06:07.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:06:07.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:06:07.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:06:07.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:06:07.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:06:07.650 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:07.645+0000 7f69eabd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:06:07.652 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:07.649+0000 7f69eabd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:06:07.654 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:07.649+0000 7f69eabd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:06:07.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:06:07.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:06:07.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:06:07.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:06:07.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:06:07.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:07.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:06:07.968 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:06:07.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:07.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:06:08.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:06:08.762 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:08.757+0000 7f69eabd97c0 -1 Falling back to public interface 2024-09-13T15:06:09.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:06:09.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:09.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:06:09.322 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:06:09.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:09.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:06:09.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:06:10.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:06:10.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:10.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:06:10.654 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:06:10.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:10.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:06:10.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:06:11.513 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:11.509+0000 7f69eabd97c0 -1 osd.1 34 log_to_monitors true 2024-09-13T15:06:11.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:06:11.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:11.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:06:11.972 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:06:11.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:11.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:06:12.485 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 39 up_thru 29 down_at 35 last_clean_interval [10,34) [v2:127.0.0.1:6810/1128607834,v1:127.0.0.1:6811/1128607834] [v2:127.0.0.1:6812/1128607834,v1:127.0.0.1:6813/1128607834] exists,up 7c0aa1ce-2183-4848-8274-f45e71e8ba57 2024-09-13T15:06:12.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:06:12.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:06:12.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:06:12.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:06:12.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:06:12.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:06:12.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:06:12.486 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:06:12.487 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:06:12.487 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:06:12.487 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:06:12.487 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:06:12.487 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:06:12.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:06:12.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:06:12.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:06:12.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:06:12.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:06:12.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:06:13.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:06:13.144 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:06:13.144 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:06:13.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:06:13.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:06:13.145 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:06:13.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-13T15:06:13.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-13T15:06:13.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-13T15:06:13.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:06:13.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:06:13.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724546 2024-09-13T15:06:13.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724546 2024-09-13T15:06:13.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-167503724546' 2024-09-13T15:06:13.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:06:13.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:06:13.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-13T15:06:13.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-13T15:06:13.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-167503724546 2-64424509452' 2024-09-13T15:06:13.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:06:13.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-13T15:06:13.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:06:13.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:06:13.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-13T15:06:13.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:06:13.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-13T15:06:13.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-13T15:06:13.664 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836506 2024-09-13T15:06:13.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:06:13.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-13T15:06:13.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:06:13.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-167503724546 2024-09-13T15:06:13.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:06:13.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:06:13.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:06:13.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-167503724546 2024-09-13T15:06:13.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724546 2024-09-13T15:06:13.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2024-09-13T15:06:13.984 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 167503724546 2024-09-13T15:06:13.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:06:14.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724546 -lt 167503724546 2024-09-13T15:06:14.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:06:14.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-13T15:06:14.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:06:14.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:06:14.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-13T15:06:14.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:06:14.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-13T15:06:14.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-13T15:06:14.322 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509452 2024-09-13T15:06:14.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:06:14.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-13T15:06:14.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:06:14.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:06:14.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:06:15.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:06:15.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:06:15.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:06:15.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:06:15.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:06:15.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:06:15.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:06:15.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:06:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T15:06:15.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:06:15.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:06:15.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:06:15.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-13T15:06:15.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' -1 2024-09-13T15:06:15.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:06:15.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=5 2024-09-13T15:06:15.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:06:15.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:06:15.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:06:15.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:06:15.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:06:15.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:06:15.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:06:15.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:06:15.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:06:15.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:06:16.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:06:16.142 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:06:16.142 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:06:16.142 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:06:16.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:06:16.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:06:16.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:06:16.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-13T15:06:16.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-09-13T15:06:16.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-09-13T15:06:16.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:06:16.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:06:16.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-09-13T15:06:16.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:06:16.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:06:16.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:06:16.949 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:06:16.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:06:17.460 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:06:17.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:06:17.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:06:17.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:06:17.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-09-13T15:06:17.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-13T15:06:17.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-13T15:06:17.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-09-13T15:06:17.703 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T15:06:17.703 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T15:06:17.703 INFO:tasks.workunit.client.0.smithi112.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000162933 s, 12.6 MB/s 2024-09-13T15:06:17.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:06:17.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:06:17.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:06:17.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:06:17.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:06:17.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:06:17.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:06:17.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:06:17.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:06:17.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:06:17.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:06:17.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:06:17.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:06:17.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:06:17.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:06:17.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:06:19.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:06:19.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:06:19.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:06:19.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:06:19.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:06:19.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:06:19.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:06:19.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:06:21.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:06:21.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:06:21.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:06:21.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:06:21.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:06:21.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:06:21.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:06:21.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:06:21.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:06:21.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:06:21.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:06:21.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:06:21.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:06:21.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:06:21.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:06:21.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:06:21.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:06:21.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:06:21.174 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:06:21.174 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:06:21.175 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:06:21.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:06:21.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:06:21.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:06:21.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:06:21.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:06:21.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:06:21.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:06:21.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:06:21.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:06:21.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:06:21.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:06:21.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:06:21.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:06:21.176 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:06:21.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:06:21.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:06:21.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:06:21.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:06:21.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:06:21.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:06:21.211 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:21.204+0000 7f6a5c1aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:06:21.214 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:21.208+0000 7f6a5c1aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:06:21.215 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:21.212+0000 7f6a5c1aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:06:21.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:06:21.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:06:21.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:06:21.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:06:21.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:06:21.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:21.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:06:21.508 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:06:21.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:21.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:06:21.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:06:22.258 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:22.252+0000 7f6a5c1aa7c0 -1 Falling back to public interface 2024-09-13T15:06:22.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:06:22.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:22.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:06:22.854 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:06:22.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:22.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:06:23.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:06:24.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:06:24.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:24.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:06:24.187 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:06:24.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:24.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:06:24.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:06:25.036 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:25.032+0000 7f6a5c1aa7c0 -1 osd.0 43 log_to_monitors true 2024-09-13T15:06:25.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:06:25.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:25.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:06:25.517 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:06:25.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:25.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:06:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 48 up_thru 48 down_at 44 last_clean_interval [5,43) [v2:127.0.0.1:6802/1998480328,v1:127.0.0.1:6803/1998480328] [v2:127.0.0.1:6804/1998480328,v1:127.0.0.1:6805/1998480328] exists,up 51e95a12-0e9f-4e04-84e7-690a88ab0006 2024-09-13T15:06:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:06:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:06:26.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:06:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:06:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:06:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:06:26.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:06:26.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:06:26.034 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:06:26.034 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:06:26.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:06:26.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:06:26.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:06:26.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:06:26.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:06:26.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:06:26.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:06:26.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:06:26.150 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:06:26.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:06:26.580 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:06:26.580 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:06:26.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:06:26.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:06:26.581 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:06:26.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=206158430210 2024-09-13T15:06:26.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 206158430210 2024-09-13T15:06:26.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-206158430210' 2024-09-13T15:06:26.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:06:26.781 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:06:26.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724550 2024-09-13T15:06:26.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724550 2024-09-13T15:06:26.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-206158430210 1-167503724550' 2024-09-13T15:06:26.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:06:26.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:06:27.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509456 2024-09-13T15:06:27.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509456 2024-09-13T15:06:27.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-206158430210 1-167503724550 2-64424509456' 2024-09-13T15:06:27.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:06:27.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-206158430210 2024-09-13T15:06:27.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:06:27.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:06:27.081 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-206158430210 2024-09-13T15:06:27.081 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:06:27.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=206158430210 2024-09-13T15:06:27.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 206158430210' 2024-09-13T15:06:27.083 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 206158430210 2024-09-13T15:06:27.083 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:06:27.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 206158430210 2024-09-13T15:06:27.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:06:28.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:06:28.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:06:28.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 206158430210 -lt 206158430210 2024-09-13T15:06:28.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:06:28.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:06:28.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-167503724550 2024-09-13T15:06:28.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:06:28.728 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-167503724550 2024-09-13T15:06:28.728 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:06:28.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724550 2024-09-13T15:06:28.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 167503724550' 2024-09-13T15:06:28.729 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 167503724550 2024-09-13T15:06:28.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:06:29.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724550 -lt 167503724550 2024-09-13T15:06:29.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:06:29.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509456 2024-09-13T15:06:29.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:06:29.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:06:29.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509456 2024-09-13T15:06:29.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:06:29.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509456 2024-09-13T15:06:29.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-13T15:06:29.070 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509456 2024-09-13T15:06:29.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:06:29.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-13T15:06:29.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:06:29.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:06:29.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:06:29.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:06:29.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:06:29.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:06:29.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:06:29.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:06:29.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:06:29.796 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:06:29.796 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:06:30.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:06:30.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:06:30.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:06:30.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:06:30.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-13T15:06:30.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-13T15:06:30.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:06:30.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-13T15:06:30.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:06:30.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:06:30.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:06:30.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:06:30.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:06:30.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:06:30.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:06:30.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:06:30.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:06:30.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:06:30.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:06:30.924 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:06:30.924 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:06:30.924 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:06:31.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-13T15:06:31.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-13T15:06:31.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:06:31.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:06:31.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:06:31.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:06:31.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:06:31.320 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:06:31.320 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:06:31.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:06:31.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:06:31.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T15:06:31.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:06:31.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T15:06:31.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:06:31.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:06:31.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:06:31.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:06:31.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:06:31.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:06:31.920 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:06:31.920 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:06:32.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:06:32.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:06:32.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:06:32.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:06:32.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-13T15:06:32.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-13T15:06:32.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:06:32.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:06:32.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:06:32.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:06:32.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:06:32.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:06:32.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:06:33.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:06:33.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:06:33.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-13T15:06:33.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:06:33.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-13T15:06:33.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:06:33.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:06:33.421 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:06:33.422 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:06:33.422 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:06:33.422 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:06:33.422 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:06:33.422 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:06:33.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:06:33.746 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:06:33.747 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:06:33.747 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:06:34.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:06:34.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:06:34.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:06:34.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-13T15:06:34.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-09-13T15:06:34.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-09-13T15:06:34.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:06:34.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:06:34.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-09-13T15:06:34.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:06:34.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:06:34.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:06:34.553 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:06:34.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:06:35.073 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:06:35.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:06:35.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:06:35.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:06:35.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-09-13T15:06:35.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-13T15:06:35.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-13T15:06:35.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-09-13T15:06:35.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:06:35.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:06:35.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:06:35.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:06:35.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-09-13T15:06:35.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:06:35.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:06:35.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:06:35.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:06:35.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:06:35.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:06:35.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:06:35.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:06:35.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:06:35.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:06:35.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:06:35.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-09-13T15:06:35.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:06:35.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:06:35.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:06:35.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:06:35.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:06:35.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-09-13T15:06:37.193 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-09-13T15:06:37.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:06:37.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:06:37.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:06:37.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:06:37.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:06:37.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:06:37.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:06:37.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:06:37.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:06:37.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:06:37.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:06:37.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:06:37.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:06:37.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:06:37.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:06:37.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:06:37.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:06:37.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:06:37.734 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:06:37.734 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:06:37.734 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:06:37.734 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:06:37.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:06:37.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:06:37.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:06:37.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:06:37.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:06:37.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:06:37.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:06:37.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:06:37.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:06:37.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:06:37.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:06:37.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:06:37.736 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:06:37.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:06:37.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:06:37.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:06:37.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:06:37.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:06:37.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:06:37.771 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:37.768+0000 7f141ccc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:06:37.774 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:37.772+0000 7f141ccc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:06:37.775 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:37.772+0000 7f141ccc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:06:38.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:06:38.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:06:38.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:06:38.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:06:38.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:06:38.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:38.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:06:38.072 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:06:38.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:38.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:06:38.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:06:39.102 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:39.100+0000 7f141ccc57c0 -1 Falling back to public interface 2024-09-13T15:06:39.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:06:39.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:39.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:06:39.395 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:06:39.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:39.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:06:39.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:06:40.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:06:40.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:40.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:06:40.718 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:06:40.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:40.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:06:41.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:06:41.722 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:41.720+0000 7f141ccc57c0 -1 osd.1 52 log_to_monitors true 2024-09-13T15:06:42.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:06:42.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:42.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:06:42.065 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:06:42.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:42.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:06:42.445 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 56 up_thru 48 down_at 53 last_clean_interval [39,52) [v2:127.0.0.1:6810/1781560703,v1:127.0.0.1:6811/1781560703] [v2:127.0.0.1:6812/1781560703,v1:127.0.0.1:6813/1781560703] exists,up 7c0aa1ce-2183-4848-8274-f45e71e8ba57 2024-09-13T15:06:42.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:06:42.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:06:42.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:06:42.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:06:42.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:06:42.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:06:42.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:06:42.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:06:42.447 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:06:42.447 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:06:42.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:06:42.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:06:42.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:06:42.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:06:42.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:06:42.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:06:42.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:06:42.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:06:42.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:06:42.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:06:42.982 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:06:42.982 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:06:42.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:06:42.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:06:42.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:06:43.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=206158430214 2024-09-13T15:06:43.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 206158430214 2024-09-13T15:06:43.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-206158430214' 2024-09-13T15:06:43.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:06:43.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:06:43.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=240518168578 2024-09-13T15:06:43.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 240518168578 2024-09-13T15:06:43.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-206158430214 1-240518168578' 2024-09-13T15:06:43.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:06:43.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:06:43.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509460 2024-09-13T15:06:43.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509460 2024-09-13T15:06:43.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-206158430214 1-240518168578 2-64424509460' 2024-09-13T15:06:43.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:06:43.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-206158430214 2024-09-13T15:06:43.596 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:06:43.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:06:43.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-206158430214 2024-09-13T15:06:43.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:06:43.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=206158430214 2024-09-13T15:06:43.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 206158430214' 2024-09-13T15:06:43.599 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 206158430214 2024-09-13T15:06:43.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:06:43.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 206158430214 -lt 206158430214 2024-09-13T15:06:43.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:06:43.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-240518168578 2024-09-13T15:06:43.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:06:43.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:06:43.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-240518168578 2024-09-13T15:06:43.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:06:43.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=240518168578 2024-09-13T15:06:43.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 240518168578' 2024-09-13T15:06:43.975 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 240518168578 2024-09-13T15:06:43.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:06:44.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 240518168578 -lt 240518168578 2024-09-13T15:06:44.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:06:44.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:06:44.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509460 2024-09-13T15:06:44.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:06:44.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:06:44.415 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509460 2024-09-13T15:06:44.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509460 2024-09-13T15:06:44.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-13T15:06:44.416 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509460 2024-09-13T15:06:44.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:06:44.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509460 -lt 64424509460 2024-09-13T15:06:44.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:06:44.734 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:06:44.734 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:06:45.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:06:45.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:06:45.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:06:45.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:06:45.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:06:45.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:06:45.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:06:45.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:06:45.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:06:45.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:06:45.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:06:45.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:06:45.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-13T15:06:45.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-13T15:06:45.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:06:45.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-13T15:06:45.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:06:45.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:06:45.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:06:45.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:06:45.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:06:45.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:06:45.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:06:45.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:06:45.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:06:45.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:06:46.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:06:46.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:06:46.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:06:46.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:06:46.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:06:46.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:06:46.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:06:46.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-13T15:06:46.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-09-13T15:06:46.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-09-13T15:06:46.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:06:46.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:06:46.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-09-13T15:06:46.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:06:46.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:06:46.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:06:47.058 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:06:47.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:06:47.560 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:06:47.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:06:47.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:06:47.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:06:47.810 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-09-13T15:06:47.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-13T15:06:47.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-13T15:06:47.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-09-13T15:06:48.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-09-13T15:06:48.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-09-13T15:06:48.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-13T15:06:48.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:06:48.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:06:48.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:06:48.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:06:48.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-13T15:06:48.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:06:48.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:06:48.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:06:48.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:06:48.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:06:48.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:06:48.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:06:48.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:06:48.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:06:48.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:06:49.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:06:49.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-13T15:06:49.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:06:49.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:06:49.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:06:49.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:06:49.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:06:49.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-13T15:06:51.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:06:51.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:06:51.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:06:51.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:06:51.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:06:51.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:06:51.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:06:51.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:06:51.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:06:51.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:06:51.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:06:51.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:06:51.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:06:51.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:06:51.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:06:51.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:06:51.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:06:51.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:06:51.529 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:06:51.529 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:06:51.529 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:06:51.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:06:51.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:06:51.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:06:51.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:06:51.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:06:51.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:06:51.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:06:51.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:06:51.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:06:51.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:06:51.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:06:51.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:06:51.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:06:51.532 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:06:51.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:06:51.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:06:51.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:06:51.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:06:51.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:06:51.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:06:51.566 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:51.564+0000 7f87c8c797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:06:51.570 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:51.568+0000 7f87c8c797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:06:51.571 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:51.568+0000 7f87c8c797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:06:51.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:06:51.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:06:51.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:06:51.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:06:51.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:06:51.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:51.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:06:51.865 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:06:51.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:51.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:06:52.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:06:52.842 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:52.840+0000 7f87c8c797c0 -1 Falling back to public interface 2024-09-13T15:06:53.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:06:53.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:53.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:06:53.195 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:06:53.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:53.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:06:53.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:06:54.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:06:54.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:54.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:06:54.512 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:06:54.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:54.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:06:54.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:06:55.480 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:06:55.480+0000 7f87c8c797c0 -1 osd.0 61 log_to_monitors true 2024-09-13T15:06:55.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:06:55.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:06:55.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:06:55.843 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:06:55.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:06:55.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:06:56.212 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 66 up_thru 57 down_at 62 last_clean_interval [48,61) [v2:127.0.0.1:6802/3499517408,v1:127.0.0.1:6803/3499517408] [v2:127.0.0.1:6804/3499517408,v1:127.0.0.1:6805/3499517408] exists,up 51e95a12-0e9f-4e04-84e7-690a88ab0006 2024-09-13T15:06:56.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:06:56.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:06:56.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:06:56.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:06:56.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:06:56.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:06:56.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:06:56.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:06:56.213 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:06:56.213 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:06:56.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:06:56.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:06:56.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:06:56.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:06:56.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:06:56.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:06:56.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:06:56.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:06:56.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:06:57.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:06:57.135 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:06:57.136 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:06:57.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:06:57.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:06:57.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:06:57.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841538 2024-09-13T15:06:57.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841538 2024-09-13T15:06:57.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841538' 2024-09-13T15:06:57.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:06:57.325 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:06:57.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=240518168582 2024-09-13T15:06:57.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 240518168582 2024-09-13T15:06:57.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841538 1-240518168582' 2024-09-13T15:06:57.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:06:57.508 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:06:57.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509464 2024-09-13T15:06:57.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509464 2024-09-13T15:06:57.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841538 1-240518168582 2-64424509464' 2024-09-13T15:06:57.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:06:57.798 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-283467841538 2024-09-13T15:06:57.798 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:06:57.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:06:57.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-283467841538 2024-09-13T15:06:57.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:06:57.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841538 2024-09-13T15:06:57.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 283467841538' 2024-09-13T15:06:57.803 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 283467841538 2024-09-13T15:06:57.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:06:58.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841538 -lt 283467841538 2024-09-13T15:06:58.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:06:58.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-240518168582 2024-09-13T15:06:58.130 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:06:58.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:06:58.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-240518168582 2024-09-13T15:06:58.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:06:58.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=240518168582 2024-09-13T15:06:58.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 240518168582' 2024-09-13T15:06:58.134 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 240518168582 2024-09-13T15:06:58.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:06:58.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 240518168582 -lt 240518168582 2024-09-13T15:06:58.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:06:58.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509464 2024-09-13T15:06:58.448 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:06:58.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:06:58.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509464 2024-09-13T15:06:58.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:06:58.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509464 2024-09-13T15:06:58.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-09-13T15:06:58.452 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509464 2024-09-13T15:06:58.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:06:58.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509463 -lt 64424509464 2024-09-13T15:06:58.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:06:59.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:06:59.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:07:00.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509464 -lt 64424509464 2024-09-13T15:07:00.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:07:00.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:07:00.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:07:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:07:00.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:07:00.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:07:00.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:07:00.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:07:00.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:07:00.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:07:00.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:07:00.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:07:00.872 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:07:00.872 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:07:00.872 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:07:01.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-13T15:07:01.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-13T15:07:01.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:07:01.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-13T15:07:01.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:07:01.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:07:01.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:07:01.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:07:01.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:07:01.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:07:01.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:07:01.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:07:01.361 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:07:01.361 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:07:01.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:07:01.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:07:01.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:07:01.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:07:02.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-13T15:07:02.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-13T15:07:02.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:07:02.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:07:02.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:07:02.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:07:02.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:07:02.081 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:07:02.081 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:07:02.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:07:02.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:07:02.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T15:07:02.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:07:02.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T15:07:02.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:07:02.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:07:02.671 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:07:02.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:07:02.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:07:02.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:07:02.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:07:02.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:07:03.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:07:03.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:07:03.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:07:03.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:07:03.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-13T15:07:03.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-13T15:07:03.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:07:03.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:07:03.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:07:03.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:07:03.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:07:03.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:07:03.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:07:03.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:07:03.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:07:03.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-13T15:07:03.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:07:03.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-13T15:07:04.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:07:04.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:07:04.195 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:07:04.195 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:07:04.195 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:07:04.195 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:07:04.196 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:07:04.196 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:07:04.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:07:04.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:07:04.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:07:04.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:07:04.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:07:04.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:07:04.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:07:04.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-13T15:07:04.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:07:04.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:07:04.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-13T15:07:04.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:07:04.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-13T15:07:04.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:07:04.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:07:04.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-13T15:07:04.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:07:04.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-13T15:07:04.890 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:07:04.890 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:07:04.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:07:04.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:07:04.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:07:05.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:07:05.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-13T15:07:05.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:07:05.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:07:05.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-13T15:07:05.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:07:05.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:07:05.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-13T15:07:07.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-13T15:07:07.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:07:07.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:07:07.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T15:07:07.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:07:07.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:07:07.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:07:07.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:07:07.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:07:07.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:07:07.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T15:07:07.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T15:07:07.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:07:07.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:07:07.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:07:07.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:07:07.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:07:07.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:07:07.625 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:07:07.625 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:07:07.625 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:07:07.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:07:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:07:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:07:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:07:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:07:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:07:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:07:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:07:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:07:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:07:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:07:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T15:07:07.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T15:07:07.627 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.2 2024-09-13T15:07:07.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:07:07.628 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-13T15:07:07.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T15:07:07.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:07:07.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:07:07.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:07:07.662 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:07.659+0000 7fa66981d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:07:07.665 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:07.663+0000 7fa66981d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:07:07.666 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:07.663+0000 7fa66981d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:07:07.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T15:07:07.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:07:07.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:07:07.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:07:07.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:07:07.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:07.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:07:07.988 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:07:07.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:07.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:07:08.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:07:09.163 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:09.159+0000 7fa66981d7c0 -1 Falling back to public interface 2024-09-13T15:07:09.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:07:09.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:09.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:07:09.356 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:07:09.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:09.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:07:09.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:07:10.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:07:10.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:10.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:07:10.682 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:07:10.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:10.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:07:11.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:07:11.971 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:11.967+0000 7fa66981d7c0 -1 osd.2 68 log_to_monitors true 2024-09-13T15:07:12.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:07:12.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:12.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:07:12.003 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:07:12.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:07:12.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:12.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:07:13.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:07:13.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:13.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:07:13.363 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:07:13.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:13.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:07:13.719 INFO:tasks.workunit.client.0.smithi112.stderr:osd.2 up in weight 1 up_from 73 up_thru 73 down_at 69 last_clean_interval [15,68) [v2:127.0.0.1:6818/1653662185,v1:127.0.0.1:6819/1653662185] [v2:127.0.0.1:6820/1653662185,v1:127.0.0.1:6821/1653662185] exists,up 53e74c87-c90b-44c8-99df-7b44aa117487 2024-09-13T15:07:13.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:07:13.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:07:13.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:07:13.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:07:13.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:07:13.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:07:13.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:07:13.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:07:13.721 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:07:13.721 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:07:13.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:07:13.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:07:13.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:07:13.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:07:13.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:07:13.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:07:13.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:07:13.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:07:13.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:07:14.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:07:14.199 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:07:14.199 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:07:14.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:07:14.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:07:14.200 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:07:14.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841542 2024-09-13T15:07:14.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841542 2024-09-13T15:07:14.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841542' 2024-09-13T15:07:14.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:07:14.364 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:07:14.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=240518168586 2024-09-13T15:07:14.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 240518168586 2024-09-13T15:07:14.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841542 1-240518168586' 2024-09-13T15:07:14.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:07:14.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:07:14.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612610 2024-09-13T15:07:14.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612610 2024-09-13T15:07:14.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841542 1-240518168586 2-313532612610' 2024-09-13T15:07:14.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:07:14.647 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-283467841542 2024-09-13T15:07:14.647 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:07:14.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:07:14.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-283467841542 2024-09-13T15:07:14.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:07:14.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841542 2024-09-13T15:07:14.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 283467841542' 2024-09-13T15:07:14.650 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 283467841542 2024-09-13T15:07:14.650 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:07:14.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841541 -lt 283467841542 2024-09-13T15:07:14.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:07:15.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:07:15.955 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:07:16.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841542 -lt 283467841542 2024-09-13T15:07:16.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:07:16.286 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-240518168586 2024-09-13T15:07:16.287 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:07:16.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:07:16.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-240518168586 2024-09-13T15:07:16.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:07:16.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=240518168586 2024-09-13T15:07:16.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 240518168586' 2024-09-13T15:07:16.291 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 240518168586 2024-09-13T15:07:16.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:07:16.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 240518168586 -lt 240518168586 2024-09-13T15:07:16.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:07:16.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-313532612610 2024-09-13T15:07:16.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:07:16.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:07:16.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-313532612610 2024-09-13T15:07:16.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:07:16.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612610 2024-09-13T15:07:16.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-09-13T15:07:16.618 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 313532612610 2024-09-13T15:07:16.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:07:16.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612610 -lt 313532612610 2024-09-13T15:07:16.960 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:07:16.960 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:07:16.960 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:07:17.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:07:17.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:07:17.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:07:17.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:07:17.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:07:17.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:07:17.320 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:07:17.320 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:07:17.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:07:17.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:07:17.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:07:17.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:07:18.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:07:18.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:07:18.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:07:18.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-09-13T15:07:18.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-13T15:07:18.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:07:18.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:07:18.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-13T15:07:18.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:07:18.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-13T15:07:18.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:07:18.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:07:18.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-13T15:07:18.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:07:18.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-13T15:07:18.039 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:07:18.039 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:07:18.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:07:18.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:07:18.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:07:18.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:07:18.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-13T15:07:18.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:07:18.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:07:18.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-13T15:07:18.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:07:18.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:07:18.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-13T15:07:20.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-13T15:07:20.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:07:20.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:07:20.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T15:07:20.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:07:20.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:07:20.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:07:20.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:07:20.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:07:20.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:07:20.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T15:07:20.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T15:07:20.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:07:20.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:07:20.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:07:20.609 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:07:20.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:07:20.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:07:20.610 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:07:20.610 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:07:20.610 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:07:20.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:07:20.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:07:20.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:07:20.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:07:20.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:07:20.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:07:20.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:07:20.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:07:20.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:07:20.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:07:20.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:07:20.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T15:07:20.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T15:07:20.611 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.2 2024-09-13T15:07:20.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:07:20.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-13T15:07:20.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T15:07:20.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:07:20.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:07:20.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:07:20.646 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:20.643+0000 7fd9e00067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:07:20.648 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:20.647+0000 7fd9e00067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:07:20.650 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:20.647+0000 7fd9e00067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:07:20.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T15:07:20.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:07:20.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:07:20.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:07:20.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:07:20.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:20.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:07:20.947 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:07:20.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:20.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:07:21.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:07:21.963 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:21.959+0000 7fd9e00067c0 -1 Falling back to public interface 2024-09-13T15:07:22.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:07:22.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:22.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:07:22.291 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:07:22.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:22.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:07:22.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:07:23.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:07:23.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:23.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:07:23.627 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:07:23.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:23.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:07:23.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:07:24.676 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:24.675+0000 7fd9e00067c0 -1 osd.2 75 log_to_monitors true 2024-09-13T15:07:24.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:07:24.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:24.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:07:24.984 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:07:24.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:24.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:07:25.353 INFO:tasks.workunit.client.0.smithi112.stderr:osd.2 up in weight 1 up_from 80 up_thru 73 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6818/449518084,v1:127.0.0.1:6819/449518084] [v2:127.0.0.1:6820/449518084,v1:127.0.0.1:6821/449518084] exists,up 53e74c87-c90b-44c8-99df-7b44aa117487 2024-09-13T15:07:25.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:07:25.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:07:25.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:07:25.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:07:25.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:07:25.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:07:25.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:07:25.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:07:25.355 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:07:25.355 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:07:25.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:07:25.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:07:25.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:07:25.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:07:25.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:07:25.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:07:25.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:07:25.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:07:25.466 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:07:25.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:07:25.838 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:07:25.839 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:07:25.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:07:25.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:07:25.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:07:25.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841546 2024-09-13T15:07:25.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841546 2024-09-13T15:07:25.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841546' 2024-09-13T15:07:25.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:07:25.992 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:07:26.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=240518168589 2024-09-13T15:07:26.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 240518168589 2024-09-13T15:07:26.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841546 1-240518168589' 2024-09-13T15:07:26.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:07:26.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:07:26.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383682 2024-09-13T15:07:26.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383682 2024-09-13T15:07:26.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841546 1-240518168589 2-343597383682' 2024-09-13T15:07:26.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:07:26.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-283467841546 2024-09-13T15:07:26.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:07:26.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:07:26.350 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-283467841546 2024-09-13T15:07:26.350 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:07:26.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841546 2024-09-13T15:07:26.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 283467841546' 2024-09-13T15:07:26.351 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 283467841546 2024-09-13T15:07:26.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:07:26.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841544 -lt 283467841546 2024-09-13T15:07:26.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:07:27.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:07:27.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:07:27.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841546 -lt 283467841546 2024-09-13T15:07:27.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:07:27.992 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-240518168589 2024-09-13T15:07:27.992 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:07:27.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:07:27.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-240518168589 2024-09-13T15:07:27.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:07:27.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=240518168589 2024-09-13T15:07:27.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 240518168589' 2024-09-13T15:07:27.995 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 240518168589 2024-09-13T15:07:27.996 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:07:28.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 240518168590 -lt 240518168589 2024-09-13T15:07:28.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:07:28.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-343597383682 2024-09-13T15:07:28.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:07:28.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:07:28.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-343597383682 2024-09-13T15:07:28.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:07:28.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383682 2024-09-13T15:07:28.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 343597383682' 2024-09-13T15:07:28.333 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 343597383682 2024-09-13T15:07:28.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:07:28.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383682 -lt 343597383682 2024-09-13T15:07:28.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:07:28.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:07:28.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:07:29.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:07:29.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:07:29.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:07:29.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:07:29.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:07:29.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:07:29.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:07:29.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:07:29.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T15:07:29.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:07:29.381 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:07:29.381 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:07:29.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-13T15:07:29.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-13T15:07:29.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:07:29.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-13T15:07:29.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:07:29.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:07:29.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:07:29.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:07:29.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:07:29.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:07:29.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:07:29.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:07:29.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:07:29.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:07:30.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T15:07:30.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:07:30.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:07:30.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:07:30.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-13T15:07:30.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-13T15:07:30.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:07:30.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:07:30.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:07:30.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:07:30.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:07:30.561 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:07:30.562 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:07:30.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:07:30.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:07:30.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T15:07:30.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:07:30.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T15:07:31.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:07:31.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:07:31.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:07:31.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:07:31.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:07:31.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:07:31.159 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:07:31.159 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:07:31.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T15:07:31.470 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:07:31.471 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:07:31.471 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:07:31.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-13T15:07:31.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-13T15:07:31.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:07:31.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:07:31.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:07:31.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:07:31.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:07:31.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:07:31.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:07:32.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:07:32.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:07:32.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-13T15:07:32.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:07:32.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-13T15:07:32.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:07:32.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:07:32.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:07:32.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:07:32.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:07:32.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:07:32.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:07:32.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:07:32.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:07:32.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:07:32.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:07:32.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:07:33.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:07:33.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:07:33.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:07:33.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-09-13T15:07:33.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-13T15:07:33.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-09-13T15:07:33.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-09-13T15:07:33.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:07:33.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:07:33.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-09-13T15:07:33.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:07:33.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:07:33.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:07:33.797 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:07:33.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:07:34.207 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:07:34.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:07:34.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:07:34.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:07:34.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-09-13T15:07:34.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-13T15:07:34.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-13T15:07:34.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-09-13T15:07:34.491 INFO:tasks.workunit.client.0.smithi112.stderr:2+0 records in 2024-09-13T15:07:34.491 INFO:tasks.workunit.client.0.smithi112.stderr:2+0 records out 2024-09-13T15:07:34.491 INFO:tasks.workunit.client.0.smithi112.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000219914 s, 18.6 MB/s 2024-09-13T15:07:34.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:07:34.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:07:34.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:07:34.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:07:34.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:07:34.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:07:34.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:07:34.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:07:34.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:07:34.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:07:34.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:07:34.494 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:07:34.494 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:07:34.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:07:34.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:07:34.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:07:35.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:07:35.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:07:35.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:07:35.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:07:35.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:07:35.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:07:35.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:07:35.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:07:37.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:07:37.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:07:37.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:07:37.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:07:37.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:07:37.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:07:37.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:07:37.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:07:37.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:07:37.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:07:37.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:07:37.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:07:37.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:07:37.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:07:37.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:07:37.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:07:37.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:07:37.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:07:37.842 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:07:37.842 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:07:37.842 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:07:37.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:07:37.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:07:37.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:07:37.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:07:37.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:07:37.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:07:37.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:07:37.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:07:37.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:07:37.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:07:37.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:07:37.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:07:37.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:07:37.844 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:07:37.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:07:37.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:07:37.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:07:37.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:07:37.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:07:37.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:07:37.880 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:37.878+0000 7f1b8bf607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:07:37.882 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:37.878+0000 7f1b8bf607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:07:37.884 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:37.882+0000 7f1b8bf607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:07:38.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:07:38.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:07:38.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:07:38.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:07:38.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:07:38.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:38.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:07:38.208 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:07:38.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:38.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:07:38.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:07:39.182 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:39.178+0000 7f1b8bf607c0 -1 Falling back to public interface 2024-09-13T15:07:39.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:07:39.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:39.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:07:39.559 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:07:39.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:39.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:07:39.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:07:40.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:07:40.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:40.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:07:40.894 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:07:40.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:40.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:07:41.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:07:42.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:07:42.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:42.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:07:42.236 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:07:42.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:42.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:07:42.430 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:42.426+0000 7f1b8bf607c0 -1 osd.1 84 log_to_monitors true 2024-09-13T15:07:42.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:07:43.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:07:43.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:43.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:07:43.570 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:07:43.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:43.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:07:44.023 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 89 up_thru 89 down_at 85 last_clean_interval [56,84) [v2:127.0.0.1:6810/1031421437,v1:127.0.0.1:6811/1031421437] [v2:127.0.0.1:6812/1031421437,v1:127.0.0.1:6813/1031421437] exists,up 7c0aa1ce-2183-4848-8274-f45e71e8ba57 2024-09-13T15:07:44.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:07:44.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:07:44.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:07:44.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:07:44.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:07:44.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:07:44.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:07:44.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:07:44.024 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:07:44.024 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:07:44.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:07:44.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:07:44.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:07:44.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:07:44.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:07:44.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:07:44.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:07:44.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:07:44.127 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:07:44.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:07:44.494 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:07:44.494 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:07:44.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:07:44.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:07:44.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:07:44.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841550 2024-09-13T15:07:44.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841550 2024-09-13T15:07:44.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841550' 2024-09-13T15:07:44.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:07:44.656 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:07:44.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=382252089346 2024-09-13T15:07:44.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 382252089346 2024-09-13T15:07:44.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841550 1-382252089346' 2024-09-13T15:07:44.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:07:44.798 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:07:44.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383687 2024-09-13T15:07:44.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383687 2024-09-13T15:07:44.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-283467841550 1-382252089346 2-343597383687' 2024-09-13T15:07:44.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:07:44.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-283467841550 2024-09-13T15:07:44.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:07:44.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:07:44.941 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-283467841550 2024-09-13T15:07:44.941 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:07:44.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841550 2024-09-13T15:07:44.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 283467841550' 2024-09-13T15:07:44.943 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 283467841550 2024-09-13T15:07:44.943 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:07:45.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841549 -lt 283467841550 2024-09-13T15:07:45.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:07:46.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:07:46.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:07:46.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841550 -lt 283467841550 2024-09-13T15:07:46.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:07:46.570 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:07:46.570 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-382252089346 2024-09-13T15:07:46.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:07:46.572 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-382252089346 2024-09-13T15:07:46.572 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:07:46.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=382252089346 2024-09-13T15:07:46.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 382252089346' 2024-09-13T15:07:46.574 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 382252089346 2024-09-13T15:07:46.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:07:46.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 382252089346 -lt 382252089346 2024-09-13T15:07:46.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:07:46.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-343597383687 2024-09-13T15:07:46.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:07:46.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:07:46.883 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-343597383687 2024-09-13T15:07:46.883 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:07:46.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383687 2024-09-13T15:07:46.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 343597383687' 2024-09-13T15:07:46.884 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 343597383687 2024-09-13T15:07:46.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:07:47.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383687 -lt 343597383687 2024-09-13T15:07:47.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:07:47.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:07:47.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:07:47.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:07:47.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:07:47.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:07:47.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:07:47.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:07:47.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:07:47.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:07:47.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:07:47.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:07:47.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:07:47.946 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:07:47.946 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:07:48.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:07:48.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:07:48.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:07:48.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-13T15:07:48.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-09-13T15:07:48.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-09-13T15:07:48.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:07:48.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:07:48.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-09-13T15:07:48.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:07:48.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:07:48.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:07:48.734 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:07:48.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:07:49.245 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:07:49.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:07:49.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:07:49.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:07:49.569 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-09-13T15:07:49.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-13T15:07:49.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-13T15:07:49.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-13T15:07:49.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:07:49.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:07:49.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:07:49.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:07:49.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-13T15:07:49.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:07:49.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:07:49.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:07:49.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:07:49.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:07:49.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:07:49.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:07:49.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:07:49.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:07:49.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:07:49.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:07:49.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-13T15:07:49.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:07:49.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:07:49.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:07:49.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:07:49.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:07:49.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-09-13T15:07:52.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:07:52.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:07:52.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:07:52.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:07:52.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:07:52.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:07:52.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:07:52.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:07:52.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:07:52.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:07:52.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:07:52.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:07:52.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:07:52.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:07:52.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:07:52.143 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:07:52.143 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:07:52.143 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:07:52.145 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:07:52.145 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:07:52.145 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:07:52.145 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:07:52.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:07:52.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:07:52.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:07:52.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:07:52.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:07:52.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:07:52.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:07:52.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:07:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:07:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:07:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:07:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:07:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:07:52.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:07:52.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:07:52.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:07:52.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:07:52.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:07:52.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:07:52.181 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:52.178+0000 7f5b5525e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:07:52.193 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:52.190+0000 7f5b5525e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:07:52.195 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:52.190+0000 7f5b5525e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:07:52.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:07:52.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:07:52.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:07:52.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:07:52.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:07:52.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:52.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:07:52.489 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:07:52.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:52.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:07:52.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:07:53.490 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:53.486+0000 7f5b5525e7c0 -1 Falling back to public interface 2024-09-13T15:07:53.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:07:53.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:53.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:07:53.827 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:07:53.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:53.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:07:54.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:07:55.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:07:55.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:55.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:07:55.157 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:07:55.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:55.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:07:55.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:07:56.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:07:56.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:56.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:07:56.491 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:07:56.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:56.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:07:56.572 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:07:56.570+0000 7f5b5525e7c0 -1 osd.0 93 log_to_monitors true 2024-09-13T15:07:56.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:07:57.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:07:57.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:07:57.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:07:57.852 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:07:57.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:07:57.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:07:58.219 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 98 up_thru 98 down_at 94 last_clean_interval [66,93) [v2:127.0.0.1:6802/3121265367,v1:127.0.0.1:6803/3121265367] [v2:127.0.0.1:6804/3121265367,v1:127.0.0.1:6805/3121265367] exists,up 51e95a12-0e9f-4e04-84e7-690a88ab0006 2024-09-13T15:07:58.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:07:58.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:07:58.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:07:58.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:07:58.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:07:58.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:07:58.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:07:58.220 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:07:58.220 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:07:58.220 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:07:58.220 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:07:58.220 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:07:58.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:07:58.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:07:58.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:07:58.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:07:58.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:07:58.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:07:58.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:07:58.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:07:58.748 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:07:58.749 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:07:58.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:07:58.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:07:58.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:07:58.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795010 2024-09-13T15:07:58.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795010 2024-09-13T15:07:58.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795010' 2024-09-13T15:07:58.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:07:58.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:07:59.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=382252089350 2024-09-13T15:07:59.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 382252089350 2024-09-13T15:07:59.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795010 1-382252089350' 2024-09-13T15:07:59.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:07:59.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:07:59.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383690 2024-09-13T15:07:59.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383690 2024-09-13T15:07:59.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795010 1-382252089350 2-343597383690' 2024-09-13T15:07:59.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:07:59.279 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-420906795010 2024-09-13T15:07:59.279 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:07:59.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:07:59.281 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-420906795010 2024-09-13T15:07:59.281 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:07:59.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795010 2024-09-13T15:07:59.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 420906795010' 2024-09-13T15:07:59.283 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 420906795010 2024-09-13T15:07:59.283 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:07:59.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 420906795010 2024-09-13T15:07:59.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:08:00.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:08:00.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:08:00.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795010 -lt 420906795010 2024-09-13T15:08:00.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:08:00.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-382252089350 2024-09-13T15:08:00.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:08:00.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:08:00.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-382252089350 2024-09-13T15:08:00.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:08:00.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=382252089350 2024-09-13T15:08:00.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 382252089350' 2024-09-13T15:08:00.941 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 382252089350 2024-09-13T15:08:00.941 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:08:01.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 382252089350 -lt 382252089350 2024-09-13T15:08:01.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:08:01.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-343597383690 2024-09-13T15:08:01.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:08:01.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:08:01.275 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-343597383690 2024-09-13T15:08:01.276 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:08:01.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383690 2024-09-13T15:08:01.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 343597383690' 2024-09-13T15:08:01.277 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 343597383690 2024-09-13T15:08:01.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:08:01.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383690 -lt 343597383690 2024-09-13T15:08:01.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:08:01.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:08:01.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:08:02.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:08:02.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:08:02.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:08:02.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:08:02.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:08:02.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:08:02.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:08:02.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:08:02.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:08:02.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:08:02.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:08:02.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:08:02.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:08:02.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:08:02.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:08:02.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-09-13T15:08:02.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-13T15:08:02.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:08:02.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:08:02.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:08:02.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:08:02.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-13T15:08:02.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:08:02.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:08:02.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:08:02.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:08:02.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:08:02.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:08:02.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:08:02.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:08:02.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:08:02.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:08:03.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:08:03.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-13T15:08:03.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:08:03.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:08:03.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:08:03.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:08:03.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:08:03.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-13T15:08:05.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:08:05.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:08:05.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:08:05.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:08:05.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:08:05.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:08:05.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:08:05.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:08:05.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:08:05.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:08:05.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:08:05.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:08:05.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:08:05.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:08:05.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:08:05.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:08:05.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:08:05.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:08:05.334 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:08:05.334 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:08:05.334 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:08:05.335 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:08:05.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:08:05.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:08:05.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:08:05.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:08:05.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:08:05.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:08:05.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:08:05.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:08:05.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:08:05.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:08:05.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:08:05.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:08:05.336 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:08:05.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:08:05.336 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:08:05.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:08:05.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:08:05.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:08:05.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:08:05.371 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:05.366+0000 7f56147a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:08:05.374 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:05.370+0000 7f56147a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:08:05.375 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:05.374+0000 7f56147a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:08:05.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:08:05.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:08:05.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:08:05.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:08:05.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:08:05.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:08:05.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:08:05.669 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:08:05.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:08:05.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:08:06.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:08:06.910 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:06.906+0000 7f56147a57c0 -1 Falling back to public interface 2024-09-13T15:08:07.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:08:07.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:08:07.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:08:07.039 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:08:07.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:08:07.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:08:07.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:08:08.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:08:08.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:08:08.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:08:08.371 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:08:08.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:08:08.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:08:08.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:08:09.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:08:09.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:08:09.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:08:09.699 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:08:09.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:08:09.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:08:09.889 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:09.886+0000 7f56147a57c0 -1 osd.1 101 log_to_monitors true 2024-09-13T15:08:10.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:08:11.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:08:11.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:08:11.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:08:11.040 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:08:11.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:08:11.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:08:11.417 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 106 up_thru 106 down_at 102 last_clean_interval [89,101) [v2:127.0.0.1:6810/4122298305,v1:127.0.0.1:6811/4122298305] [v2:127.0.0.1:6812/4122298305,v1:127.0.0.1:6813/4122298305] exists,up 7c0aa1ce-2183-4848-8274-f45e71e8ba57 2024-09-13T15:08:11.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:08:11.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:08:11.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:08:11.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:08:11.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:08:11.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:08:11.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:08:11.420 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:08:11.421 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:08:11.421 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:08:11.421 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:08:11.421 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:08:11.421 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:08:11.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:08:11.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:08:11.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:08:11.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:08:11.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:08:11.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:08:12.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:08:12.069 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:08:12.069 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:08:12.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:08:12.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:08:12.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:08:12.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795014 2024-09-13T15:08:12.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795014 2024-09-13T15:08:12.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795014' 2024-09-13T15:08:12.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:08:12.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:08:12.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533378 2024-09-13T15:08:12.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533378 2024-09-13T15:08:12.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795014 1-455266533378' 2024-09-13T15:08:12.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:08:12.367 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:08:12.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383694 2024-09-13T15:08:12.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383694 2024-09-13T15:08:12.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795014 1-455266533378 2-343597383694' 2024-09-13T15:08:12.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:08:12.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-420906795014 2024-09-13T15:08:12.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:08:12.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:08:12.512 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-420906795014 2024-09-13T15:08:12.512 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:08:12.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795014 2024-09-13T15:08:12.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 420906795014' 2024-09-13T15:08:12.514 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 420906795014 2024-09-13T15:08:12.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:08:12.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795012 -lt 420906795014 2024-09-13T15:08:12.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:08:13.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:08:13.836 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:08:14.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795014 -lt 420906795014 2024-09-13T15:08:14.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:08:14.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-455266533378 2024-09-13T15:08:14.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:08:14.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:08:14.155 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-455266533378 2024-09-13T15:08:14.156 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:08:14.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533378 2024-09-13T15:08:14.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 455266533378' 2024-09-13T15:08:14.157 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 455266533378 2024-09-13T15:08:14.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:08:14.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533378 -lt 455266533378 2024-09-13T15:08:14.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:08:14.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-343597383694 2024-09-13T15:08:14.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:08:14.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:08:14.482 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-343597383694 2024-09-13T15:08:14.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:08:14.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383694 2024-09-13T15:08:14.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 343597383694' 2024-09-13T15:08:14.488 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 343597383694 2024-09-13T15:08:14.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:08:14.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383694 -lt 343597383694 2024-09-13T15:08:14.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:08:14.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:08:14.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:08:15.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:08:15.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:08:15.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:08:15.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:08:15.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:08:15.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:08:15.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:08:15.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:08:15.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:08:15.550 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:08:15.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:08:15.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:08:15.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:08:15.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:08:15.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:08:15.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-13T15:08:15.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-09-13T15:08:15.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-09-13T15:08:15.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:08:15.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:08:15.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-09-13T15:08:15.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:08:15.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:08:15.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:08:16.414 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:08:16.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:08:16.856 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:08:16.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:08:16.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:08:16.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:08:17.088 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-09-13T15:08:17.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-13T15:08:17.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-13T15:08:17.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-13T15:08:17.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-13T15:08:17.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-09-13T15:08:17.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-13T15:08:17.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:08:17.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:08:17.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:08:17.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:08:17.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-13T15:08:17.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:08:17.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:08:17.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:08:17.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:08:17.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:08:17.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:08:17.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:08:17.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:08:17.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:08:17.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:08:17.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:08:17.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-13T15:08:17.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:08:17.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:08:17.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:08:17.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:08:17.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:08:17.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-09-13T15:08:19.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:08:19.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:08:19.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:08:19.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:08:19.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:08:19.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:08:19.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:08:19.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:08:19.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:08:19.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:08:19.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:08:19.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:08:19.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:08:19.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:08:19.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:08:19.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:08:19.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:08:19.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:08:19.810 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:08:19.810 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:08:19.810 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:08:19.810 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:08:19.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:08:19.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:08:19.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:08:19.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:08:19.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:08:19.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:08:19.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:08:19.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:08:19.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:08:19.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:08:19.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:08:19.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:08:19.811 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:08:19.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:08:19.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:08:19.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:08:19.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:08:19.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:08:19.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:08:19.845 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:19.841+0000 7fea907697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:08:19.847 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:19.841+0000 7fea907697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:08:19.849 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:19.845+0000 7fea907697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:08:20.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:08:20.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:08:20.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:08:20.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:08:20.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:08:20.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:08:20.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:08:20.149 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:08:20.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:08:20.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:08:20.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:08:21.166 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:21.161+0000 7fea907697c0 -1 Falling back to public interface 2024-09-13T15:08:21.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:08:21.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:08:21.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:08:21.486 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:08:21.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:08:21.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:08:21.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:08:22.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:08:22.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:08:22.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:08:22.824 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:08:22.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:08:22.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:08:23.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:08:23.929 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:23.925+0000 7fea907697c0 -1 osd.0 110 log_to_monitors true 2024-09-13T15:08:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:08:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:08:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:08:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:08:24.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:08:24.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:08:24.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:08:25.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:08:25.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:08:25.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:08:25.495 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:08:25.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:08:25.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:08:25.955 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 115 up_thru 115 down_at 111 last_clean_interval [98,110) [v2:127.0.0.1:6802/945829966,v1:127.0.0.1:6803/945829966] [v2:127.0.0.1:6804/945829966,v1:127.0.0.1:6805/945829966] exists,up 51e95a12-0e9f-4e04-84e7-690a88ab0006 2024-09-13T15:08:25.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:08:25.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:08:25.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:08:25.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:08:25.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:08:25.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:08:25.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:08:25.956 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:08:25.956 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:08:25.957 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:08:25.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:08:25.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:08:25.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:08:26.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:08:26.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:08:26.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:08:26.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:08:26.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:08:26.068 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:08:26.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:08:26.449 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:08:26.449 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:08:26.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:08:26.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:08:26.449 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:08:26.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=493921239042 2024-09-13T15:08:26.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 493921239042 2024-09-13T15:08:26.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-493921239042' 2024-09-13T15:08:26.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:08:26.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:08:26.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533382 2024-09-13T15:08:26.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533382 2024-09-13T15:08:26.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-493921239042 1-455266533382' 2024-09-13T15:08:26.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:08:26.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:08:26.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383698 2024-09-13T15:08:26.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383698 2024-09-13T15:08:26.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-493921239042 1-455266533382 2-343597383698' 2024-09-13T15:08:26.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:08:26.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-493921239042 2024-09-13T15:08:26.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:08:26.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:08:26.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-493921239042 2024-09-13T15:08:26.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:08:26.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=493921239042 2024-09-13T15:08:26.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 493921239042' 2024-09-13T15:08:26.913 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 493921239042 2024-09-13T15:08:26.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:08:27.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 493921239042 2024-09-13T15:08:27.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:08:28.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:08:28.236 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:08:28.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 493921239042 -lt 493921239042 2024-09-13T15:08:28.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:08:28.570 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-455266533382 2024-09-13T15:08:28.570 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:08:28.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:08:28.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-455266533382 2024-09-13T15:08:28.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:08:28.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533382 2024-09-13T15:08:28.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 455266533382' 2024-09-13T15:08:28.575 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 455266533382 2024-09-13T15:08:28.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:08:28.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533382 -lt 455266533382 2024-09-13T15:08:28.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:08:28.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-343597383698 2024-09-13T15:08:28.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:08:28.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:08:28.921 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-343597383698 2024-09-13T15:08:28.922 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:08:28.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383698 2024-09-13T15:08:28.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 343597383698' 2024-09-13T15:08:28.923 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 343597383698 2024-09-13T15:08:28.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:08:29.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383698 -lt 343597383698 2024-09-13T15:08:29.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:08:29.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:08:29.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:08:29.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:08:29.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:08:29.671 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:08:29.671 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:08:29.671 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:08:29.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:08:29.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:08:29.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:08:30.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:08:30.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:08:30.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:08:30.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:08:30.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:08:30.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:08:30.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:08:30.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-13T15:08:30.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:08:30.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:08:30.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:08:30.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:08:30.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-13T15:08:30.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:08:30.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:08:30.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:08:30.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:08:30.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:08:30.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:08:30.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:08:30.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:08:30.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:08:30.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:08:30.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:08:30.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-13T15:08:30.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:08:30.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:08:30.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:08:30.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:08:30.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:08:30.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-13T15:08:33.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:08:33.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:08:33.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:08:33.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:08:33.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:08:33.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:08:33.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:08:33.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:08:33.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:08:33.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:08:33.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:08:33.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:08:33.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:08:33.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:08:33.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:08:33.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:08:33.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:08:33.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:08:33.395 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:08:33.395 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:08:33.395 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:08:33.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:08:33.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:08:33.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:08:33.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:08:33.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:08:33.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:08:33.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:08:33.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:08:33.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:08:33.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:08:33.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:08:33.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:08:33.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:08:33.397 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:08:33.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:08:33.397 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:08:33.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:08:33.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:08:33.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:08:33.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:08:33.431 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:33.425+0000 7f6ec24757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:08:33.433 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:33.429+0000 7f6ec24757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:08:33.435 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:33.429+0000 7f6ec24757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:08:33.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:08:33.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:08:33.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:08:33.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:08:33.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:08:33.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:08:33.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:08:33.725 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:08:33.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:08:33.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:08:34.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:08:34.786 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:34.781+0000 7f6ec24757c0 -1 Falling back to public interface 2024-09-13T15:08:35.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:08:35.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:08:35.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:08:35.143 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:08:35.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:08:35.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:08:35.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:08:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:08:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:08:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:08:36.471 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:08:36.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:08:36.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:08:36.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:08:37.485 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:37.481+0000 7f6ec24757c0 -1 osd.0 117 log_to_monitors true 2024-09-13T15:08:37.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:08:37.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:08:37.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:08:37.806 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:08:37.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:08:37.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:08:38.339 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 122 up_thru 115 down_at 118 last_clean_interval [115,117) [v2:127.0.0.1:6802/4288681289,v1:127.0.0.1:6803/4288681289] [v2:127.0.0.1:6804/4288681289,v1:127.0.0.1:6805/4288681289] exists,up 51e95a12-0e9f-4e04-84e7-690a88ab0006 2024-09-13T15:08:38.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:08:38.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:08:38.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:08:38.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:08:38.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:08:38.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:08:38.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:08:38.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:08:38.341 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:08:38.341 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:08:38.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:08:38.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:08:38.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:08:38.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:08:38.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:08:38.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:08:38.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:08:38.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:08:38.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:08:38.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:08:38.932 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:08:38.932 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:08:38.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:08:38.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:08:38.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:08:39.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=523986010114 2024-09-13T15:08:39.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 523986010114 2024-09-13T15:08:39.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-523986010114' 2024-09-13T15:08:39.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:08:39.123 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:08:39.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533385 2024-09-13T15:08:39.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533385 2024-09-13T15:08:39.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-523986010114 1-455266533385' 2024-09-13T15:08:39.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:08:39.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:08:39.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383701 2024-09-13T15:08:39.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383701 2024-09-13T15:08:39.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-523986010114 1-455266533385 2-343597383701' 2024-09-13T15:08:39.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:08:39.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-523986010114 2024-09-13T15:08:39.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:08:39.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:08:39.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-523986010114 2024-09-13T15:08:39.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:08:39.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=523986010114 2024-09-13T15:08:39.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 523986010114' 2024-09-13T15:08:39.409 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 523986010114 2024-09-13T15:08:39.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:08:39.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 523986010114 2024-09-13T15:08:39.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:08:40.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:08:40.735 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:08:41.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 523986010114 -lt 523986010114 2024-09-13T15:08:41.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:08:41.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-455266533385 2024-09-13T15:08:41.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:08:41.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:08:41.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:08:41.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-455266533385 2024-09-13T15:08:41.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533385 2024-09-13T15:08:41.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 455266533385' 2024-09-13T15:08:41.074 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 455266533385 2024-09-13T15:08:41.075 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:08:41.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533385 -lt 455266533385 2024-09-13T15:08:41.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:08:41.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-343597383701 2024-09-13T15:08:41.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:08:41.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:08:41.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:08:41.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-343597383701 2024-09-13T15:08:41.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383701 2024-09-13T15:08:41.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 343597383701' 2024-09-13T15:08:41.407 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 343597383701 2024-09-13T15:08:41.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:08:41.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383701 -lt 343597383701 2024-09-13T15:08:41.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:08:41.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:08:41.734 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:08:42.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:08:42.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:08:42.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:08:42.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:08:42.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:08:42.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:08:42.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:08:42.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:08:42.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:08:42.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:08:42.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:08:42.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:08:42.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:08:42.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:08:42.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:08:42.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-09-13T15:08:42.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-09-13T15:08:42.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T15:08:42.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=EOBJ0 2024-09-13T15:08:42.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool EOBJ0 2024-09-13T15:08:42.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T15:08:43.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-09-13T15:08:43.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-09-13T15:08:43.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=3.0 2024-09-13T15:08:43.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 3.0 2024-09-13T15:08:43.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:08:43.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:08:43.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:08:43.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:08:43.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-13T15:05:52.353780+0000 2024-09-13T15:08:43.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 3.0 2024-09-13T15:08:43.785 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-09-13T15:08:43.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 3.0 2024-09-13T15:05:52.353780+0000 2024-09-13T15:08:43.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-13T15:08:43.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:05:52.353780+0000 2024-09-13T15:08:43.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:08:43.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:08:43.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:08:43.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-13T15:08:43.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:08:43.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:08:43.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:08:43.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:08:44.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:05:52.353780+0000 '>' 2024-09-13T15:05:52.353780+0000 2024-09-13T15:08:44.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:08:45.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:08:45.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:08:45.130 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-13T15:08:45.130 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:08:45.130 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:08:45.130 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:08:45.131 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:08:45.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:05:52.353780+0000 '>' 2024-09-13T15:05:52.353780+0000 2024-09-13T15:08:45.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:08:46.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:08:46.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:08:46.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-13T15:08:46.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:08:46.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:08:46.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:08:46.458 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:08:46.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:08:44.206744+0000 '>' 2024-09-13T15:05:52.353780+0000 2024-09-13T15:08:46.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:08:46.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-13T15:08:46.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-13T15:08:46.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-09-13T15:08:46.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-13T15:08:46.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-13T15:08:46.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-13T15:08:46.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-13T15:08:46.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=122 2024-09-13T15:08:46.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-13T15:08:46.925 INFO:tasks.workunit.client.0.smithi112.stderr: . as $in 2024-09-13T15:08:46.925 INFO:tasks.workunit.client.0.smithi112.stderr: | if type == "object" then 2024-09-13T15:08:46.925 INFO:tasks.workunit.client.0.smithi112.stderr: reduce keys[] as $key 2024-09-13T15:08:46.925 INFO:tasks.workunit.client.0.smithi112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-13T15:08:46.925 INFO:tasks.workunit.client.0.smithi112.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-13T15:08:46.925 INFO:tasks.workunit.client.0.smithi112.stderr: else f 2024-09-13T15:08:46.925 INFO:tasks.workunit.client.0.smithi112.stderr: end; 2024-09-13T15:08:46.925 INFO:tasks.workunit.client.0.smithi112.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-13T15:08:46.925 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-13T15:08:46.925 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-13T15:08:46.925 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-13T15:08:46.926 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-13T15:08:46.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-09-13T15:08:46.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-13T15:08:46.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-13T15:08:46.966 INFO:tasks.workunit.client.0.smithi112.stderr: . as $in 2024-09-13T15:08:46.966 INFO:tasks.workunit.client.0.smithi112.stderr: | if type == "object" then 2024-09-13T15:08:46.966 INFO:tasks.workunit.client.0.smithi112.stderr: reduce keys[] as $key 2024-09-13T15:08:46.966 INFO:tasks.workunit.client.0.smithi112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-13T15:08:46.966 INFO:tasks.workunit.client.0.smithi112.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-13T15:08:46.966 INFO:tasks.workunit.client.0.smithi112.stderr: else f 2024-09-13T15:08:46.966 INFO:tasks.workunit.client.0.smithi112.stderr: end; 2024-09-13T15:08:46.966 INFO:tasks.workunit.client.0.smithi112.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-13T15:08:46.967 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-13T15:08:46.967 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-13T15:08:46.967 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-13T15:08:46.967 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-13T15:08:46.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-09-13T15:08:46.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-13T15:08:47.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-13T15:08:47.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-13T15:08:47.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-09-13T15:08:47.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-09-13T15:08:47.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-09-13T15:08:47.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-13T15:08:47.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:08:47.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:08:47.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:08:47.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:08:47.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:08:47.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-13T15:05:52.353780+0000 2024-09-13T15:08:47.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-13T15:08:47.613 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-09-13T15:08:47.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-13T15:05:52.353780+0000 last_deep_scrub_stamp 2024-09-13T15:08:47.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-13T15:08:47.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:05:52.353780+0000 2024-09-13T15:08:47.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-13T15:08:47.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:08:47.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:08:47.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:08:47.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:08:47.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:08:47.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:08:47.631 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:08:47.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:05:52.353780+0000 '>' 2024-09-13T15:05:52.353780+0000 2024-09-13T15:08:47.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:08:48.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:08:48.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:08:48.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:08:48.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:08:48.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:08:48.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:08:48.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:08:49.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:05:52.353780+0000 '>' 2024-09-13T15:05:52.353780+0000 2024-09-13T15:08:49.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:08:50.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:08:50.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:08:50.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:08:50.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:08:50.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:08:50.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:08:50.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:08:50.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:05:52.353780+0000 '>' 2024-09-13T15:05:52.353780+0000 2024-09-13T15:08:50.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:08:51.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:08:51.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:08:51.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:08:51.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:08:51.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:08:51.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:08:51.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:08:51.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:08:48.310920+0000 '>' 2024-09-13T15:05:52.353780+0000 2024-09-13T15:08:51.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:08:51.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-13T15:08:51.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-13T15:08:52.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-09-13T15:08:52.008 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-13T15:08:52.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-13T15:08:52.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-13T15:08:52.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-13T15:08:52.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=122 2024-09-13T15:08:52.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-09-13T15:08:52.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-13T15:08:52.096 INFO:tasks.workunit.client.0.smithi112.stderr: . as $in 2024-09-13T15:08:52.096 INFO:tasks.workunit.client.0.smithi112.stderr: | if type == "object" then 2024-09-13T15:08:52.096 INFO:tasks.workunit.client.0.smithi112.stderr: reduce keys[] as $key 2024-09-13T15:08:52.096 INFO:tasks.workunit.client.0.smithi112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-13T15:08:52.096 INFO:tasks.workunit.client.0.smithi112.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-13T15:08:52.096 INFO:tasks.workunit.client.0.smithi112.stderr: else f 2024-09-13T15:08:52.096 INFO:tasks.workunit.client.0.smithi112.stderr: end; 2024-09-13T15:08:52.096 INFO:tasks.workunit.client.0.smithi112.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-13T15:08:52.097 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-13T15:08:52.097 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-13T15:08:52.097 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-13T15:08:52.097 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-13T15:08:52.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-09-13T15:08:52.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-13T15:08:52.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-13T15:08:52.139 INFO:tasks.workunit.client.0.smithi112.stderr: . as $in 2024-09-13T15:08:52.139 INFO:tasks.workunit.client.0.smithi112.stderr: | if type == "object" then 2024-09-13T15:08:52.140 INFO:tasks.workunit.client.0.smithi112.stderr: reduce keys[] as $key 2024-09-13T15:08:52.140 INFO:tasks.workunit.client.0.smithi112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-13T15:08:52.140 INFO:tasks.workunit.client.0.smithi112.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-13T15:08:52.140 INFO:tasks.workunit.client.0.smithi112.stderr: else f 2024-09-13T15:08:52.140 INFO:tasks.workunit.client.0.smithi112.stderr: end; 2024-09-13T15:08:52.140 INFO:tasks.workunit.client.0.smithi112.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-13T15:08:52.140 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-13T15:08:52.140 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-13T15:08:52.140 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-13T15:08:52.140 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-13T15:08:52.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-09-13T15:08:52.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-13T15:08:52.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-13T15:08:52.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-13T15:08:52.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-09-13T15:08:52.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-09-13T15:08:52.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-09-13T15:08:52.587 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'ecpool' removed 2024-09-13T15:08:52.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T15:08:52.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:08:52.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:08:52.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:08:52.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:08:52.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:08:52.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:08:52.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:08:52.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:08:52.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:08:52.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:08:52.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:08:52.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:08:52.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:08:52.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:08:52.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:08:52.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:08:52.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:08:52.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:08:52.931 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:08:52.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:08:52.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:08:52.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:08:52.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:08:52.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:08:52.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:08:52.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:08:52.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:08:52.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:08:52.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T15:08:52.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T15:08:52.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T15:08:52.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T15:08:52.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:08:52.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:08:52.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:08:52.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:08:52.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:08:52.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:08:52.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:08:52.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:08:52.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:08:52.976 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:08:52.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:08:52.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:08:52.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:08:52.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:08:52.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:08:52.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:08:52.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:08:52.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:08:52.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:08:52.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:08:52.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:08:52.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:08:52.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:08:52.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:08:52.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:08:52.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:08:52.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:08:52.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:08:52.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T15:08:52.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:08:52.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:08:52.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:08:52.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T15:08:52.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:08:52.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T15:08:52.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:08:52.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T15:08:52.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-09-13T15:08:52.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-09-13T15:08:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-09-13T15:08:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-09-13T15:08:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-13T15:08:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T15:08:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:08:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:08:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:08:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T15:08:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-13T15:08:53.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:08:53.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:08:53.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:08:53.441 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:08:53.441 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:08:53.441 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:08:53.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:08:53.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-13T15:08:53.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:08:53.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:08:53.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:08:53.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:08:53.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:08:53.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:08:53.638 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:08:53.638 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:08:53.639 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:08:53.639 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:08:53.639 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:08:53.640 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:08:53.640 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:08:53.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:08:53.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T15:08:53.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:08:53.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:08:53.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:08:53.751 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:08:53.751 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:08:53.751 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:08:53.751 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:08:53.751 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:08:53.752 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:08:53.752 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:08:53.752 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:08:53.752 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:08:53.752 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:08:53.753 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T15:08:53.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-09-13T15:08:53.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T15:08:53.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:08:53.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:08:53.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:08:53.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T15:08:53.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:08:54.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:08:54.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:08:54.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:08:54.709 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:08:54.709 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:08:54.709 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:08:54.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:08:54.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:08:54.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:08:54.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-09-13T15:08:54.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:08:54.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:08:54.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:08:54.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:08:54.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:08:54.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:08:54.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:08:54.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:08:54.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:08:54.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:08:54.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:08:54.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:08:54.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:08:54.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:08:54.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:08:54.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:08:54.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:08:54.748 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:08:54.748 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:08:54.749 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:08:54.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:08:54.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:08:54.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:08:54.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:08:54.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:08:54.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:08:54.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:08:54.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:08:54.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:08:54.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:08:54.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:08:54.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:08:54.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:08:54.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:08:54.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:08:54.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2e31414c-0818-44ce-b14a-578b74fc94f5 2024-09-13T15:08:54.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2e31414c-0818-44ce-b14a-578b74fc94f5' 2024-09-13T15:08:54.751 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 2e31414c-0818-44ce-b14a-578b74fc94f5 2024-09-13T15:08:54.751 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:08:54.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCGVeRm88vALRAAWHalaG2ILgxo07raNXN+tA== 2024-09-13T15:08:54.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCGVeRm88vALRAAWHalaG2ILgxo07raNXN+tA=="}' 2024-09-13T15:08:54.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2e31414c-0818-44ce-b14a-578b74fc94f5 -i td/osd-scrub-repair/0/new.json 2024-09-13T15:08:55.004 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:08:55.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T15:08:55.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCGVeRm88vALRAAWHalaG2ILgxo07raNXN+tA== --osd-uuid 2e31414c-0818-44ce-b14a-578b74fc94f5 2024-09-13T15:08:55.060 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:55.056+0000 7f4294cad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:08:55.063 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:55.056+0000 7f4294cad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:08:55.064 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:55.060+0000 7f4294cad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:08:55.064 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:55.060+0000 7f4294cad7c0 -1 bdev(0x558da78efc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:08:55.065 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:08:55.060+0000 7f4294cad7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T15:09:01.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T15:09:01.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:09:01.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:09:01.421 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:09:01.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:09:01.813 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:09:01.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:09:01.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:09:01.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:09:01.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:09:01.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:09:01.847 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:09:01.840+0000 7f9ef44247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:09:01.878 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:09:01.872+0000 7f9ef44247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:09:01.879 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:09:01.872+0000 7f9ef44247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:09:02.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:09:02.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:09:02.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:09:02.148 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:09:02.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:09:02.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:09:02.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:02.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:09:02.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:02.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:02.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:03.394 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:09:03.388+0000 7f9ef44247c0 -1 Falling back to public interface 2024-09-13T15:09:03.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:03.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:03.487 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:09:03.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:09:03.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:03.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:03.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:04.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:04.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:04.805 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:09:04.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:09:04.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:04.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:05.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:06.118 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:09:06.112+0000 7f9ef44247c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:09:06.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:06.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:06.132 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:09:06.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:09:06.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:06.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:06.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:07.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:07.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:07.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:09:07.558 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:09:07.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:07.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:07.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:08.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:08.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:08.900 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:09:08.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:09:08.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:08.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:09.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:10.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:10.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:10.231 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:09:10.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:09:10.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:10.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:10.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:11.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:11.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:11.569 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:09:11.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:09:11.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:11.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:11.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:12.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:12.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:12.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:09:12.898 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:09:12.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:12.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:13.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:14.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:14.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:14.234 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:09:14.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:09:14.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:14.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:14.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:15.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:15.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:15.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:09:15.574 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:09:15.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:15.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:15.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:16.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:16.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:16.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:09:16.902 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:09:16.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:16.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:17.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:18.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:18.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:18.236 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:09:18.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:09:18.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:18.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:18.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:19.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:19.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:19.951 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:09:19.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:09:19.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:19.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:20.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:21.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:21.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:21.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:09:21.551 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:09:21.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:21.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:22.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:23.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:23.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:23.201 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:09:23.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:09:23.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:23.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:23.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:24.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:24.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:24.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:09:24.675 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:09:24.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:24.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:25.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:26.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:26.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:26.415 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:09:26.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:09:26.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:26.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:27.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:28.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:28.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:28.232 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:09:28.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:09:28.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:28.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:09:28.605 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1884751332,v1:127.0.0.1:6803/1884751332] [v2:127.0.0.1:6804/1884751332,v1:127.0.0.1:6805/1884751332] exists,up 2e31414c-0818-44ce-b14a-578b74fc94f5 2024-09-13T15:09:28.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:09:28.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:09:28.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:09:28.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-09-13T15:09:28.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:09:28.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:09:28.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T15:09:28.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:09:28.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:09:28.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:09:28.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:09:28.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:09:28.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:09:28.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:09:28.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:09:28.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:09:28.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:09:28.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:09:28.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:09:28.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:09:28.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:09:28.610 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:09:28.610 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:09:28.610 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:09:28.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:09:28.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:09:28.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:09:28.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:09:28.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:09:28.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:09:28.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:09:28.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:09:28.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:09:28.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:09:28.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:09:28.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:09:28.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:09:28.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:09:28.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:09:28.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:09:28.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f2c8290b-0073-4005-a233-4452b6605eb6' 2024-09-13T15:09:28.612 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:09:28.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:09:28.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCoVeRmEA10JRAAKKvKfKl2V0pnXuPcLwG24w== 2024-09-13T15:09:28.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCoVeRmEA10JRAAKKvKfKl2V0pnXuPcLwG24w=="}' 2024-09-13T15:09:28.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f2c8290b-0073-4005-a233-4452b6605eb6 -i td/osd-scrub-repair/1/new.json 2024-09-13T15:09:29.084 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:09:29.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T15:09:29.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCoVeRmEA10JRAAKKvKfKl2V0pnXuPcLwG24w== --osd-uuid f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:09:29.144 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:09:29.144+0000 7f991ff717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:09:29.160 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:09:29.160+0000 7f991ff717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:09:29.162 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:09:29.160+0000 7f991ff717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:09:29.162 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:09:29.160+0000 7f991ff717c0 -1 bdev(0x55fb6c8efc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T15:09:29.163 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:09:29.160+0000 7f991ff717c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T15:09:35.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T15:09:35.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:09:35.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T15:09:35.220 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T15:09:35.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:09:35.629 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:09:35.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T15:09:35.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:09:35.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:09:35.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:09:35.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:09:35.661 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:09:35.659+0000 7f544edcd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:09:35.664 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:09:35.663+0000 7f544edcd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:09:35.665 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:09:35.663+0000 7f544edcd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:09:35.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T15:09:35.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:09:35.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:09:35.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:09:35.969 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:09:35.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:09:35.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:35.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:09:35.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:35.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:36.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:37.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:37.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:37.317 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:09:37.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:09:37.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:37.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:37.409 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:09:37.407+0000 7f544edcd7c0 -1 Falling back to public interface 2024-09-13T15:09:37.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:38.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:38.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:38.652 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:09:38.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:09:38.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:38.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:38.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:39.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:39.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:39.973 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:09:39.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:09:39.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:39.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:40.161 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:09:40.155+0000 7f544edcd7c0 -1 osd.1 0 log_to_monitors true 2024-09-13T15:09:40.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:41.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:41.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:41.385 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:09:41.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:09:41.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:41.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:41.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:42.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:42.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:42.720 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:09:42.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:09:42.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:42.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:43.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:44.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:44.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:44.221 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:09:44.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:09:44.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:44.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:44.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:45.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:45.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:45.608 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:09:45.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:09:45.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:45.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:45.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:46.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:46.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:46.973 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:09:46.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:09:46.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:46.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:47.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:48.305 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:09:48.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:09:48.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:48.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:48.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:49.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:49.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:49.630 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:09:49.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:09:49.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:49.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:50.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:51.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:51.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:51.041 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:09:51.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:09:51.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:51.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:51.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:52.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:52.370 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:09:52.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:52.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:09:52.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:52.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:52.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:53.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:53.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:53.709 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:09:53.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:09:53.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:53.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:54.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:55.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:55.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:55.086 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:09:55.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:09:55.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:55.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:55.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:56.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:56.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:56.513 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:09:56.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:09:56.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:56.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:57.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:58.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:58.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:58.387 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:09:58.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:09:58.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:58.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:09:58.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:09:59.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:09:59.836 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:09:59.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:09:59.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:09:59.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:09:59.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:10:00.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:10:01.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:10:01.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:10:01.371 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:10:01.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:10:01.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:10:01.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:10:02.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:10:03.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:10:03.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:10:03.121 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:10:03.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:10:03.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:10:03.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:10:03.603 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2000384352,v1:127.0.0.1:6811/2000384352] [v2:127.0.0.1:6812/2000384352,v1:127.0.0.1:6813/2000384352] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:10:03.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:10:03.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:10:03.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:10:03.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-09-13T15:10:03.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T15:10:03.922 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' does not exist 2024-09-13T15:10:03.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T15:10:03.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T15:10:04.335 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T15:10:04.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:10:05.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T15:10:07.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-09-13T15:10:07.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:10:07.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:10:07.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:10:07.433 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:10:07.434 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:10:07.434 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:10:07.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:10:07.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:10:07.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:10:07.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:10:07.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:10:07.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:10:07.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:10:07.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:10:07.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:10:07.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:10:07.939 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:10:07.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:10:07.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:10:07.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:10:08.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836493 2024-09-13T15:10:08.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836493 2024-09-13T15:10:08.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836493' 2024-09-13T15:10:08.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:10:08.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:10:08.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672966 2024-09-13T15:10:08.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672966 2024-09-13T15:10:08.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836493 1-42949672966' 2024-09-13T15:10:08.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:10:08.202 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836493 2024-09-13T15:10:08.202 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:10:08.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:10:08.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836493 2024-09-13T15:10:08.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:10:08.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836493 2024-09-13T15:10:08.205 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836493 2024-09-13T15:10:08.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-09-13T15:10:08.206 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:10:08.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836493 2024-09-13T15:10:08.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:10:09.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:10:09.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:10:09.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836493 2024-09-13T15:10:09.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:10:09.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672966 2024-09-13T15:10:09.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:10:09.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:10:09.856 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672966 2024-09-13T15:10:09.856 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:10:09.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672966 2024-09-13T15:10:09.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-09-13T15:10:09.857 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672966 2024-09-13T15:10:09.857 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:10:10.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672967 -lt 42949672966 2024-09-13T15:10:10.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:10:10.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:10:10.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:10:10.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T15:10:10.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:10:10.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:10:10.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:10:10.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:10:10.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:10:10.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:10:10.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:10:10.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T15:10:10.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:10:10.924 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:10:10.924 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:10:11.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T15:10:11.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:10:11.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:10:11.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-09-13T15:10:11.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2024-09-13T15:10:11.731 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'foo' created 2024-09-13T15:10:11.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:10:12.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-09-13T15:10:12.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2024-09-13T15:10:13.212 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'csr_pool' created 2024-09-13T15:10:13.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:10:14.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-09-13T15:10:14.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:10:14.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:10:14.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:10:14.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:10:14.240 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:10:14.240 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:10:14.240 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:10:14.240 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:10:14.240 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:10:14.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:10:14.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:10:14.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:10:14.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:10:14.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:10:14.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:10:14.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:10:14.772 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:10:14.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:10:14.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:10:14.773 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:10:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836496 2024-09-13T15:10:14.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836496 2024-09-13T15:10:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836496' 2024-09-13T15:10:14.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:10:14.931 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:10:15.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672969 2024-09-13T15:10:15.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672969 2024-09-13T15:10:15.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836496 1-42949672969' 2024-09-13T15:10:15.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:10:15.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836496 2024-09-13T15:10:15.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:10:15.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:10:15.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836496 2024-09-13T15:10:15.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:10:15.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836496 2024-09-13T15:10:15.074 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836496 2024-09-13T15:10:15.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-09-13T15:10:15.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:10:15.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836496 2024-09-13T15:10:15.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:10:16.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:10:16.390 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:10:16.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836496 -lt 21474836496 2024-09-13T15:10:16.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:10:16.728 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672969 2024-09-13T15:10:16.728 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:10:16.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:10:16.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672969 2024-09-13T15:10:16.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:10:16.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672969 2024-09-13T15:10:16.732 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672969 2024-09-13T15:10:16.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-13T15:10:16.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:10:17.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672969 -lt 42949672969 2024-09-13T15:10:17.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:10:17.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:10:17.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:10:17.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:10:17.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:10:17.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:10:17.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:10:17.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:10:17.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:10:17.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:10:17.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:10:17.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:10:17.810 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:10:17.810 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:10:17.810 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:10:18.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:10:18.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:10:18.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:10:18.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-09-13T15:10:18.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:18.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-09-13T15:10:18.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-09-13T15:10:18.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:18.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:18.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-09-13T15:10:18.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:18.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:18.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:18.615 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:18.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:19.092 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:19.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:19.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:19.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:19.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-09-13T15:10:19.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-09-13T15:10:19.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:19.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-09-13T15:10:19.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-09-13T15:10:19.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:19.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:19.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-09-13T15:10:19.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:19.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:19.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:20.113 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:20.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:20.564 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:20.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:20.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:20.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:20.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-09-13T15:10:20.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-09-13T15:10:21.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:21.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-09-13T15:10:21.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-09-13T15:10:21.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:21.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:21.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-09-13T15:10:21.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:21.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:21.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:21.509 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:21.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:21.960 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:21.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:21.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:21.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:22.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-09-13T15:10:22.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-09-13T15:10:22.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:22.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-09-13T15:10:22.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-09-13T15:10:22.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:22.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:22.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-09-13T15:10:22.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:22.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:22.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:22.972 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:22.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:23.507 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:23.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:23.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:23.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:23.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-09-13T15:10:23.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-09-13T15:10:23.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:23.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-09-13T15:10:23.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-09-13T15:10:23.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:23.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:23.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-09-13T15:10:23.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:23.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:23.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:24.418 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:24.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:24.937 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:24.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:24.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:24.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:25.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-09-13T15:10:25.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-09-13T15:10:25.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:25.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-09-13T15:10:25.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-09-13T15:10:25.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:25.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:25.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-09-13T15:10:25.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:25.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:25.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:25.957 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:25.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:26.400 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:26.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:26.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:26.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:26.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-09-13T15:10:26.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-09-13T15:10:26.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:26.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-09-13T15:10:26.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-09-13T15:10:26.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:26.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:26.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-09-13T15:10:26.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:26.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:26.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:27.361 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:27.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:27.880 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:27.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:27.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:27.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:28.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-09-13T15:10:28.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-09-13T15:10:28.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:28.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-09-13T15:10:28.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-09-13T15:10:28.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:28.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:28.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-09-13T15:10:28.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:28.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:28.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:28.791 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:28.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:29.318 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:29.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:29.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:29.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:29.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-09-13T15:10:29.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-09-13T15:10:29.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:29.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-09-13T15:10:29.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-09-13T15:10:29.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:29.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:29.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-09-13T15:10:29.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:29.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:29.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:30.332 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:30.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:30.843 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:30.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:30.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:30.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:31.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-09-13T15:10:31.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-09-13T15:10:31.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:31.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-09-13T15:10:31.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-09-13T15:10:31.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:31.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:31.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-09-13T15:10:31.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:31.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:31.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:31.871 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:31.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:32.331 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:32.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:32.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:32.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:32.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-09-13T15:10:32.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-09-13T15:10:32.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:32.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-09-13T15:10:32.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-09-13T15:10:32.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:32.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:32.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-09-13T15:10:32.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:32.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:32.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:33.251 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:33.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:33.678 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:33.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:33.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:33.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:33.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-09-13T15:10:34.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-09-13T15:10:34.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:34.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-09-13T15:10:34.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-09-13T15:10:34.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:34.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:34.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-09-13T15:10:34.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:34.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:34.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:34.589 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:34.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:35.192 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:35.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:35.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:35.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:35.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-09-13T15:10:35.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-09-13T15:10:35.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:35.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-09-13T15:10:35.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-09-13T15:10:35.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:35.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:35.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-09-13T15:10:35.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:35.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:35.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:36.120 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:36.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:36.606 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:36.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:36.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:36.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:36.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-09-13T15:10:37.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-09-13T15:10:37.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:37.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-09-13T15:10:37.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-09-13T15:10:37.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:37.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:37.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-09-13T15:10:37.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:37.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:37.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:37.584 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:37.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:38.069 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:38.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:38.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:38.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:38.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-09-13T15:10:38.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-09-13T15:10:38.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:38.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-09-13T15:10:38.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-09-13T15:10:38.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:38.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:38.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-09-13T15:10:38.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:38.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:38.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:39.072 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:39.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:39.616 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:39.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:39.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:39.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:39.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-09-13T15:10:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-09-13T15:10:40.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:40.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-09-13T15:10:40.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-09-13T15:10:40.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:40.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:40.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-09-13T15:10:40.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:40.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:40.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:40.578 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:40.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:41.097 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:41.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:41.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:41.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:41.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-09-13T15:10:41.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-09-13T15:10:41.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:41.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-09-13T15:10:41.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-09-13T15:10:41.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:41.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:41.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-09-13T15:10:41.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:41.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:41.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:42.134 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:42.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:42.585 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:42.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:42.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:42.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:42.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-09-13T15:10:43.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-09-13T15:10:43.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:43.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-09-13T15:10:43.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-09-13T15:10:43.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:43.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:43.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-09-13T15:10:43.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:43.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:43.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:43.555 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:43.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:44.057 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:44.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:44.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:44.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:44.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-09-13T15:10:44.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-09-13T15:10:44.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:44.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-09-13T15:10:44.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-09-13T15:10:44.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:10:44.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:10:44.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-09-13T15:10:44.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:10:44.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:10:44.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:10:45.027 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:10:45.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:10:45.512 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:10:45.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:10:45.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:10:45.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:10:45.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-09-13T15:10:45.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-09-13T15:10:46.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-09-13T15:10:46.098 INFO:tasks.workunit.client.0.smithi112.stderr:1025+0 records in 2024-09-13T15:10:46.099 INFO:tasks.workunit.client.0.smithi112.stderr:1025+0 records out 2024-09-13T15:10:46.099 INFO:tasks.workunit.client.0.smithi112.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.003465 s, 303 MB/s 2024-09-13T15:10:46.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-09-13T15:10:46.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-09-13T15:10:46.306 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-09-13T15:10:46.306 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2024-09-13T15:10:46.307 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2024-09-13T15:10:46.307 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-09-13T15:10:46.307 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T15:10:46.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-09-13T15:10:46.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-09-13T15:10:46.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2024-09-13T15:10:46.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2024-09-13T15:10:46.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-09-13T15:10:46.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:10:46.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-09-13T15:10:46.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-09-13T15:10:46.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T15:10:46.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T15:10:46.985 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:10:46.985 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:10:46.985 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:10:46.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.0.asok 2024-09-13T15:10:46.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-09-13T15:10:46.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.63282/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-09-13T15:10:47.081 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:10:47.081 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-09-13T15:10:47.081 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:10:47.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-09-13T15:10:47.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T15:10:47.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T15:10:47.094 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:10:47.094 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:10:47.094 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:10:47.095 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.1.asok 2024-09-13T15:10:47.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-09-13T15:10:47.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.63282/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-09-13T15:10:47.192 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:10:47.192 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-09-13T15:10:47.192 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:10:47.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-09-13T15:10:47.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-13T15:10:47.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:10:47.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:10:47.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:10:47.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:10:47.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:10:47.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-13T15:10:13.132175+0000 2024-09-13T15:10:47.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-13T15:10:47.797 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-09-13T15:10:47.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-13T15:10:13.132175+0000 last_deep_scrub_stamp 2024-09-13T15:10:47.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-13T15:10:47.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:10:13.132175+0000 2024-09-13T15:10:47.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-13T15:10:47.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:10:47.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:10:47.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:10:47.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:10:47.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:10:47.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:10:47.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:10:48.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:10:13.132175+0000 '>' 2024-09-13T15:10:13.132175+0000 2024-09-13T15:10:48.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:10:49.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:10:49.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:10:49.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:10:49.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:10:49.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:10:49.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:10:49.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:10:49.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:10:13.132175+0000 '>' 2024-09-13T15:10:13.132175+0000 2024-09-13T15:10:49.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:10:50.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:10:50.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:10:50.485 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:10:50.486 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:10:50.486 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:10:50.486 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:10:50.486 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:10:50.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:10:48.448335+0000 '>' 2024-09-13T15:10:13.132175+0000 2024-09-13T15:10:50.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:10:50.821 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-09-13T15:10:50.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:10:50.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-09-13T15:10:50.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-09-13T15:10:50.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-13T15:10:50.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:10:50.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-09-13T15:10:50.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-09-13T15:10:50.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:10:50.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:10:50.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:10:50.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:10:50.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:10:50.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:10:50.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:10:50.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:10:50.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:10:50.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:10:50.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:10:50.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:10:50.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:10:50.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:10:50.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:10:50.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:10:51.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:10:51.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:10:51.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:10:51.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:10:51.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:10:51.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:10:51.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:10:51.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:10:52.774 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-09-13T15:10:53.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:10:53.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:10:53.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:10:53.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:10:53.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:10:53.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:10:53.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:10:53.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:10:53.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:10:53.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:10:53.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:10:53.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:10:53.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:10:53.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:10:53.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:10:53.344 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:10:53.344 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:10:53.344 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:10:53.345 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:10:53.345 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:10:53.345 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:10:53.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:10:53.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:10:53.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:10:53.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:10:53.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:10:53.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:10:53.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:10:53.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:10:53.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:10:53.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:10:53.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:10:53.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:10:53.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:10:53.347 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:10:53.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:10:53.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:10:53.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:10:53.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:10:53.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:10:53.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:10:53.382 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:10:53.378+0000 7fbc9fcd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:10:53.385 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:10:53.382+0000 7fbc9fcd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:10:53.387 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:10:53.382+0000 7fbc9fcd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:10:53.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:10:53.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:10:53.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:10:53.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:10:53.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:10:53.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:10:53.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:10:53.681 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:10:53.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:10:53.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:10:54.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:10:54.922 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:10:54.917+0000 7fbc9fcd37c0 -1 Falling back to public interface 2024-09-13T15:10:55.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:10:55.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:10:55.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:10:55.027 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:10:55.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:10:55.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:10:55.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:10:56.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:10:56.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:10:56.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:10:56.369 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:10:56.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:10:56.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:10:56.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:10:57.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:10:57.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:10:57.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:10:57.704 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:10:57.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:10:57.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:10:58.104 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:10:58.101+0000 7fbc9fcd37c0 -1 osd.1 64 log_to_monitors true 2024-09-13T15:10:58.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:10:59.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:10:59.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:10:59.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:10:59.165 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:10:59.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:10:59.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:10:59.641 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [10,64) [v2:127.0.0.1:6810/2924152596,v1:127.0.0.1:6811/2924152596] [v2:127.0.0.1:6812/2924152596,v1:127.0.0.1:6813/2924152596] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:10:59.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:10:59.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:10:59.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:10:59.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:10:59.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:10:59.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:10:59.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:10:59.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:10:59.642 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:10:59.642 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:10:59.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:10:59.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:10:59.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:10:59.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:10:59.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:10:59.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:10:59.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:10:59.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:10:59.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:11:00.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:11:00.077 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:11:00.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:11:00.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:11:00.078 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:11:00.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-13T15:11:00.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-13T15:11:00.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-13T15:11:00.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:11:00.224 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:11:00.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776130 2024-09-13T15:11:00.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776130 2024-09-13T15:11:00.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-292057776130' 2024-09-13T15:11:00.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:11:00.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-13T15:11:00.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:11:00.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:11:00.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-13T15:11:00.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:11:00.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-13T15:11:00.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-13T15:11:00.358 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836506 2024-09-13T15:11:00.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:11:00.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836506 2024-09-13T15:11:00.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:11:01.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:11:01.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:11:01.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-13T15:11:01.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:11:01.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-292057776130 2024-09-13T15:11:01.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:11:01.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:11:01.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-292057776130 2024-09-13T15:11:01.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:11:01.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776130 2024-09-13T15:11:01.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 292057776130' 2024-09-13T15:11:01.976 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 292057776130 2024-09-13T15:11:01.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:11:02.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776130 -lt 292057776130 2024-09-13T15:11:02.315 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:11:02.315 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:11:02.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:11:02.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:11:02.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:11:02.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:11:02.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:11:02.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:11:02.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:11:02.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:11:02.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:11:03.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:11:03.064 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:11:03.065 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:11:03.065 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:11:03.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:11:03.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:11:03.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:11:03.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:11:03.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-09-13T15:11:03.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-09-13T15:11:03.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-13T15:11:03.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:11:03.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-09-13T15:11:03.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-09-13T15:11:03.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:11:03.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:11:03.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:11:03.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:11:03.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:11:03.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:11:03.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:11:03.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:11:03.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:11:03.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:11:03.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:11:03.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:11:03.442 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:11:03.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:11:03.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:11:03.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:11:03.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:11:03.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:11:03.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:11:03.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:11:03.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:11:03.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:11:03.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:11:03.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-13T15:11:05.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:11:05.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:11:05.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:11:05.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:11:05.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:11:05.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:11:05.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:11:05.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:11:05.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:11:05.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:11:05.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:11:05.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:11:05.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:11:05.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:11:05.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:11:05.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:11:05.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:11:05.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:11:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:11:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:11:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:11:05.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:11:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:11:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:11:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:11:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:11:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:11:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:11:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:11:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:11:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:11:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:11:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:11:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:11:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:11:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:11:05.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:11:05.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:11:05.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:11:05.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:11:05.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:11:05.916 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:05.913+0000 7fc6aa5ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:11:05.919 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:05.913+0000 7fc6aa5ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:11:05.920 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:05.917+0000 7fc6aa5ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:11:06.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:11:06.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:11:06.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:11:06.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:11:06.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:11:06.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:06.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:11:06.211 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:11:06.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:06.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:11:06.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:11:06.978 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:06.973+0000 7fc6aa5ea7c0 -1 Falling back to public interface 2024-09-13T15:11:07.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:11:07.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:07.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:11:07.542 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:11:07.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:07.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:11:07.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:11:08.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:11:08.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:08.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:11:08.863 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:11:08.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:08.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:11:09.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:11:09.852 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:09.849+0000 7fc6aa5ea7c0 -1 osd.0 70 log_to_monitors true 2024-09-13T15:11:10.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:11:10.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:10.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:11:10.176 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:11:10.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:10.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:11:10.513 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 74 up_thru 69 down_at 71 last_clean_interval [5,70) [v2:127.0.0.1:6802/819592877,v1:127.0.0.1:6803/819592877] [v2:127.0.0.1:6804/819592877,v1:127.0.0.1:6805/819592877] exists,up 2e31414c-0818-44ce-b14a-578b74fc94f5 2024-09-13T15:11:10.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:11:10.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:11:10.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:11:10.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:11:10.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:11:10.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:11:10.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:11:10.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:11:10.515 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:11:10.515 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:11:10.515 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:11:10.515 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:11:10.515 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:11:10.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:11:10.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:11:10.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:11:10.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:11:10.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:11:10.626 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:11:11.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:11:11.024 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:11:11.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:11:11.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:11:11.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:11:11.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=317827579906 2024-09-13T15:11:11.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 317827579906 2024-09-13T15:11:11.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-317827579906' 2024-09-13T15:11:11.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:11:11.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:11:11.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776133 2024-09-13T15:11:11.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776133 2024-09-13T15:11:11.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-317827579906 1-292057776133' 2024-09-13T15:11:11.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:11:11.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-317827579906 2024-09-13T15:11:11.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:11:11.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:11:11.325 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-317827579906 2024-09-13T15:11:11.325 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:11:11.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=317827579906 2024-09-13T15:11:11.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 317827579906' 2024-09-13T15:11:11.326 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 317827579906 2024-09-13T15:11:11.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:11:11.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 317827579906 2024-09-13T15:11:11.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:11:12.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:11:12.632 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:11:12.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 317827579906 -lt 317827579906 2024-09-13T15:11:12.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:11:12.952 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-292057776133 2024-09-13T15:11:12.952 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:11:12.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:11:12.955 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-292057776133 2024-09-13T15:11:12.955 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:11:12.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776133 2024-09-13T15:11:12.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 292057776133' 2024-09-13T15:11:12.956 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 292057776133 2024-09-13T15:11:12.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:11:13.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776133 -lt 292057776133 2024-09-13T15:11:13.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:11:13.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:11:13.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:11:13.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:11:13.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:11:13.697 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:11:13.697 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:11:13.697 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:11:13.697 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:11:13.697 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:11:13.697 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:11:14.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:11:14.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:11:14.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:11:14.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:11:14.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:11:14.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:11:14.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:11:14.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:11:14.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-09-13T15:11:14.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-09-13T15:11:14.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-13T15:11:14.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:11:14.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-09-13T15:11:14.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:11:14.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:11:14.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:11:14.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:11:14.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-09-13T15:11:14.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:11:14.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:11:14.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:11:14.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:11:14.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:11:14.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:11:14.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:11:14.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:11:14.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:11:14.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:11:14.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:11:14.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-09-13T15:11:14.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:11:14.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:11:14.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:11:14.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:11:14.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:11:14.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-09-13T15:11:16.378 INFO:tasks.workunit.client.0.smithi112.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-09-13T15:11:16.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:11:16.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:11:16.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:11:16.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:11:16.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:11:16.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:11:16.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:11:16.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:11:16.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:11:16.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:11:16.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:11:16.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:11:16.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:11:16.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:11:16.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:11:16.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:11:16.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:11:16.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:11:16.918 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:11:16.918 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:11:16.918 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:11:16.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:11:16.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:11:16.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:11:16.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:11:16.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:11:16.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:11:16.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:11:16.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:11:16.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:11:16.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:11:16.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:11:16.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:11:16.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:11:16.919 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:11:16.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:11:16.920 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:11:16.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:11:16.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:11:16.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:11:16.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:11:16.958 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:16.953+0000 7febb02ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:11:16.961 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:16.957+0000 7febb02ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:11:16.962 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:16.957+0000 7febb02ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:11:17.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:11:17.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:11:17.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:11:17.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:11:17.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:11:17.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:17.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:11:17.252 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:11:17.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:17.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:11:17.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:11:18.230 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:18.225+0000 7febb02ca7c0 -1 Falling back to public interface 2024-09-13T15:11:18.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:11:18.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:18.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:11:18.631 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:11:18.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:18.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:11:18.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:11:19.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:11:19.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:19.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:11:19.969 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:11:19.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:19.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:11:20.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:11:20.918 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:20.913+0000 7febb02ca7c0 -1 osd.1 75 log_to_monitors true 2024-09-13T15:11:21.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:11:21.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:21.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:11:21.293 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:11:21.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:21.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:11:21.929 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 79 up_thru 79 down_at 76 last_clean_interval [68,75) [v2:127.0.0.1:6810/2627001199,v1:127.0.0.1:6811/2627001199] [v2:127.0.0.1:6812/2627001199,v1:127.0.0.1:6813/2627001199] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:11:21.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:11:21.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:11:21.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:11:21.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:11:21.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:11:21.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:11:21.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:11:21.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:11:21.930 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:11:21.930 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:11:21.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:11:21.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:11:21.931 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:11:22.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:11:22.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:11:22.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:11:22.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:11:22.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:11:22.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:11:22.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:11:22.382 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:11:22.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:11:22.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:11:22.383 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:11:22.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=317827579909 2024-09-13T15:11:22.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 317827579909 2024-09-13T15:11:22.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-317827579909' 2024-09-13T15:11:22.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:11:22.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:11:22.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=339302416386 2024-09-13T15:11:22.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 339302416386 2024-09-13T15:11:22.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-317827579909 1-339302416386' 2024-09-13T15:11:22.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:11:22.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-317827579909 2024-09-13T15:11:22.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:11:22.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:11:22.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-317827579909 2024-09-13T15:11:22.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:11:22.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=317827579909 2024-09-13T15:11:22.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 317827579909' 2024-09-13T15:11:22.687 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 317827579909 2024-09-13T15:11:22.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:11:23.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 317827579908 -lt 317827579909 2024-09-13T15:11:23.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:11:24.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:11:24.012 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:11:24.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 317827579909 -lt 317827579909 2024-09-13T15:11:24.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:11:24.350 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-339302416386 2024-09-13T15:11:24.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:11:24.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:11:24.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-339302416386 2024-09-13T15:11:24.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:11:24.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=339302416386 2024-09-13T15:11:24.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 339302416386' 2024-09-13T15:11:24.355 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 339302416386 2024-09-13T15:11:24.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:11:24.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416386 -lt 339302416386 2024-09-13T15:11:24.693 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:11:24.694 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:11:24.694 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:11:25.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:11:25.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:11:25.102 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:11:25.102 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:11:25.102 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:11:25.102 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:11:25.102 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:11:25.103 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:11:25.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:11:25.439 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:11:25.439 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:11:25.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:11:25.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:11:25.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:11:25.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:11:25.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:11:25.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-09-13T15:11:25.835 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-09-13T15:11:25.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-13T15:11:25.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:11:25.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-13T15:11:25.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:11:25.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:11:25.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:11:25.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:11:25.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-13T15:11:25.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:11:25.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:11:25.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:11:25.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:11:25.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:11:25.837 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:11:25.837 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:11:25.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:11:25.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:11:25.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:11:26.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:11:26.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-13T15:11:26.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:11:26.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:11:26.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:11:26.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:11:26.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:11:26.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-13T15:11:28.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:11:28.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:11:28.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:11:28.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:11:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:11:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:11:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:11:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:11:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:11:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:11:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:11:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:11:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:11:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:11:28.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:11:28.080 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:11:28.080 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:11:28.081 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:11:28.082 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:11:28.082 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:11:28.082 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:11:28.082 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:11:28.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:11:28.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:11:28.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:11:28.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:11:28.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:11:28.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:11:28.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:11:28.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:11:28.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:11:28.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:11:28.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:11:28.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:11:28.083 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:11:28.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:11:28.084 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:11:28.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:11:28.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:11:28.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:11:28.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:11:28.122 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:28.117+0000 7fc802baa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:11:28.125 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:28.121+0000 7fc802baa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:11:28.127 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:28.121+0000 7fc802baa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:11:28.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:11:28.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:11:28.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:11:28.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:11:28.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:11:28.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:28.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:11:28.424 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:11:28.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:28.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:11:28.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:11:29.170 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:29.165+0000 7fc802baa7c0 -1 Falling back to public interface 2024-09-13T15:11:29.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:11:29.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:29.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:11:29.767 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:11:29.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:29.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:11:30.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:11:31.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:11:31.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:31.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:11:31.128 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:11:31.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:31.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:11:31.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:11:31.957 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:31.952+0000 7fc802baa7c0 -1 osd.0 80 log_to_monitors true 2024-09-13T15:11:32.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:11:32.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:32.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:11:32.457 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:11:32.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:32.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:11:32.799 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [74,80) [v2:127.0.0.1:6802/3358972429,v1:127.0.0.1:6803/3358972429] [v2:127.0.0.1:6804/3358972429,v1:127.0.0.1:6805/3358972429] exists,up 2e31414c-0818-44ce-b14a-578b74fc94f5 2024-09-13T15:11:32.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:11:32.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:11:32.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:11:32.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:11:32.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:11:32.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:11:32.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:11:32.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:11:32.801 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:11:32.801 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:11:32.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:11:32.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:11:32.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:11:32.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:11:32.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:11:32.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:11:32.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:11:32.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:11:32.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:11:33.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:11:33.318 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:11:33.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:11:33.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:11:33.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:11:33.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252866 2024-09-13T15:11:33.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252866 2024-09-13T15:11:33.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-360777252866' 2024-09-13T15:11:33.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:11:33.509 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:11:33.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=339302416389 2024-09-13T15:11:33.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 339302416389 2024-09-13T15:11:33.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-360777252866 1-339302416389' 2024-09-13T15:11:33.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:11:33.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-360777252866 2024-09-13T15:11:33.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:11:33.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:11:33.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-360777252866 2024-09-13T15:11:33.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:11:33.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252866 2024-09-13T15:11:33.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 360777252866' 2024-09-13T15:11:33.645 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 360777252866 2024-09-13T15:11:33.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:11:33.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252866 -lt 360777252866 2024-09-13T15:11:33.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:11:33.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-339302416389 2024-09-13T15:11:33.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:11:33.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:11:33.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-339302416389 2024-09-13T15:11:33.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:11:33.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=339302416389 2024-09-13T15:11:33.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 339302416389' 2024-09-13T15:11:33.953 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 339302416389 2024-09-13T15:11:33.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:11:34.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416388 -lt 339302416389 2024-09-13T15:11:34.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:11:35.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:11:35.279 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:11:35.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416388 -lt 339302416389 2024-09-13T15:11:35.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:11:36.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T15:11:36.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:11:36.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416389 -lt 339302416389 2024-09-13T15:11:36.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:11:36.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:11:36.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:11:37.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:11:37.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:11:37.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:11:37.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:11:37.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:11:37.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:11:37.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:11:37.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:11:37.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:11:37.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:11:37.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:11:37.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:11:38.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:11:38.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:11:38.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:11:38.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:11:38.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-09-13T15:11:38.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-09-13T15:11:38.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-13T15:11:38.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:11:38.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-09-13T15:11:38.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:11:38.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:11:38.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:11:38.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:11:38.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-09-13T15:11:38.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:11:38.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:11:38.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:11:38.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:11:38.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:11:38.038 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:11:38.038 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:11:38.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:11:38.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:11:38.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:11:38.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:11:38.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-09-13T15:11:38.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:11:38.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:11:38.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:11:38.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:11:38.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:11:38.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-09-13T15:11:40.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:11:40.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:11:40.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:11:40.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:11:40.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:11:40.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:11:40.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:11:40.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:11:40.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:11:40.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:11:40.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:11:40.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:11:40.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:11:40.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:11:40.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:11:40.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:11:40.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:11:40.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:11:40.387 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:11:40.387 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:11:40.387 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:11:40.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:11:40.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:11:40.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:11:40.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:11:40.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:11:40.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:11:40.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:11:40.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:11:40.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:11:40.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:11:40.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:11:40.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:11:40.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:11:40.389 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:11:40.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:11:40.390 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:11:40.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:11:40.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:11:40.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:11:40.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:11:40.424 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:40.420+0000 7f9169da27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:11:40.426 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:40.420+0000 7f9169da27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:11:40.428 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:40.424+0000 7f9169da27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:11:40.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:11:40.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:11:40.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:11:40.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:11:40.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:11:40.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:40.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:11:40.729 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:11:40.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:40.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:11:41.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:11:41.498 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:41.492+0000 7f9169da27c0 -1 Falling back to public interface 2024-09-13T15:11:42.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:11:42.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:42.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:11:42.073 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:11:42.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:42.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:11:42.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:11:43.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:11:43.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:43.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:11:43.409 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:11:43.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:43.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:11:43.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:11:44.260 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:44.260+0000 7f9169da27c0 -1 osd.1 86 log_to_monitors true 2024-09-13T15:11:44.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:11:44.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:44.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:11:44.750 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:11:44.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:44.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:11:45.196 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 90 up_thru 85 down_at 87 last_clean_interval [79,86) [v2:127.0.0.1:6810/2646158040,v1:127.0.0.1:6811/2646158040] [v2:127.0.0.1:6812/2646158040,v1:127.0.0.1:6813/2646158040] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:11:45.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:11:45.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:11:45.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:11:45.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:11:45.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:11:45.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:11:45.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:11:45.197 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:11:45.198 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:11:45.198 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:11:45.198 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:11:45.198 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:11:45.198 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:11:45.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:11:45.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:11:45.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:11:45.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:11:45.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:11:45.315 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:11:45.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:11:45.771 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:11:45.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:11:45.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:11:45.772 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:11:45.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252869 2024-09-13T15:11:45.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252869 2024-09-13T15:11:45.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-360777252869' 2024-09-13T15:11:45.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:11:45.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:11:46.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=386547056642 2024-09-13T15:11:46.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 386547056642 2024-09-13T15:11:46.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-360777252869 1-386547056642' 2024-09-13T15:11:46.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:11:46.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-360777252869 2024-09-13T15:11:46.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:11:46.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:11:46.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-360777252869 2024-09-13T15:11:46.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:11:46.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252869 2024-09-13T15:11:46.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 360777252869' 2024-09-13T15:11:46.075 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 360777252869 2024-09-13T15:11:46.075 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:11:46.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252868 -lt 360777252869 2024-09-13T15:11:46.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:11:47.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:11:47.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:11:47.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252870 -lt 360777252869 2024-09-13T15:11:47.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:11:47.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:11:47.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-386547056642 2024-09-13T15:11:47.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:11:47.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-386547056642 2024-09-13T15:11:47.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:11:47.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=386547056642 2024-09-13T15:11:47.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2024-09-13T15:11:47.778 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 386547056642 2024-09-13T15:11:47.779 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:11:48.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 386547056642 -lt 386547056642 2024-09-13T15:11:48.128 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:11:48.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:11:48.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:11:48.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:11:48.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:11:48.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:11:48.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:11:48.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:11:48.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:11:48.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:11:48.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:11:48.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:11:48.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:11:48.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:11:48.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:11:49.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:11:49.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:11:49.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:11:49.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:11:49.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-09-13T15:11:49.232 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-09-13T15:11:49.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-13T15:11:49.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:11:49.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-09-13T15:11:49.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-13T15:11:49.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:11:49.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:11:49.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:11:49.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:11:49.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-13T15:11:49.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:11:49.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:11:49.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:11:49.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:11:49.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:11:49.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:11:49.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:11:49.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:11:49.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:11:49.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:11:49.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:11:49.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-13T15:11:49.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:11:49.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:11:49.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:11:49.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:11:49.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:11:49.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-13T15:11:51.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:11:51.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:11:51.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:11:51.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:11:51.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:11:51.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:11:51.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:11:51.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:11:51.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:11:51.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:11:51.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:11:51.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:11:51.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:11:51.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:11:51.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:11:51.501 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:11:51.501 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:11:51.501 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:11:51.501 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:11:51.501 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:11:51.501 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:11:51.502 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:11:51.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:11:51.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:11:51.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:11:51.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:11:51.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:11:51.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:11:51.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:11:51.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:11:51.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:11:51.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:11:51.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:11:51.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:11:51.503 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:11:51.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:11:51.504 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:11:51.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:11:51.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:11:51.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:11:51.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:11:51.538 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:51.536+0000 7f98a80bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:11:51.540 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:51.540+0000 7f98a80bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:11:51.542 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:51.540+0000 7f98a80bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:11:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:11:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:11:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:11:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:11:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:11:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:11:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:11:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:51.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:11:52.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:11:52.818 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:52.816+0000 7f98a80bf7c0 -1 Falling back to public interface 2024-09-13T15:11:53.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:11:53.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:53.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:11:53.165 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:11:53.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:53.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:11:53.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:11:54.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:11:54.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:54.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:11:54.493 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:11:54.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:54.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:11:54.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:11:55.521 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:11:55.520+0000 7f98a80bf7c0 -1 osd.0 91 log_to_monitors true 2024-09-13T15:11:55.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:11:55.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:11:55.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:11:55.817 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:11:55.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:11:55.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:11:56.173 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 95 up_thru 90 down_at 92 last_clean_interval [84,91) [v2:127.0.0.1:6802/2903029597,v1:127.0.0.1:6803/2903029597] [v2:127.0.0.1:6804/2903029597,v1:127.0.0.1:6805/2903029597] exists,up 2e31414c-0818-44ce-b14a-578b74fc94f5 2024-09-13T15:11:56.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:11:56.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:11:56.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:11:56.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:11:56.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:11:56.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:11:56.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:11:56.174 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:11:56.175 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:11:56.175 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:11:56.176 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:11:56.176 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:11:56.176 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:11:56.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:11:56.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:11:56.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:11:56.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:11:56.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:11:56.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:11:56.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:11:56.713 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:11:56.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:11:56.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:11:56.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:11:56.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=408021893122 2024-09-13T15:11:56.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 408021893122 2024-09-13T15:11:56.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-408021893122' 2024-09-13T15:11:56.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:11:56.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:11:57.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=386547056645 2024-09-13T15:11:57.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 386547056645 2024-09-13T15:11:57.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-408021893122 1-386547056645' 2024-09-13T15:11:57.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:11:57.037 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-408021893122 2024-09-13T15:11:57.037 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:11:57.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:11:57.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-408021893122 2024-09-13T15:11:57.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:11:57.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=408021893122 2024-09-13T15:11:57.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 408021893122' 2024-09-13T15:11:57.041 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 408021893122 2024-09-13T15:11:57.042 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:11:57.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252870 -lt 408021893122 2024-09-13T15:11:57.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:11:58.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:11:58.367 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:11:58.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893122 -lt 408021893122 2024-09-13T15:11:58.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:11:58.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-386547056645 2024-09-13T15:11:58.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:11:58.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:11:58.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-386547056645 2024-09-13T15:11:58.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:11:58.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=386547056645 2024-09-13T15:11:58.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 386547056645' 2024-09-13T15:11:58.704 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 386547056645 2024-09-13T15:11:58.704 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:11:59.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 386547056645 -lt 386547056645 2024-09-13T15:11:59.033 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:11:59.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:11:59.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:11:59.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:11:59.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:11:59.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:11:59.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:11:59.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:11:59.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:11:59.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:11:59.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:11:59.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T15:11:59.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:11:59.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:11:59.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:12:00.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 6 2024-09-13T15:12:00.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T15:12:00.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:12:00.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T15:12:00.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:12:00.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:12:00.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:12:00.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:12:00.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:12:00.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:12:00.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:12:00.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:12:00.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:12:00.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:12:00.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T15:12:00.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:12:00.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:12:00.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:12:00.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 6 2024-09-13T15:12:00.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T15:12:00.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:12:00.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:12:00.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:12:00.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:12:00.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:12:00.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:12:00.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:12:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:12:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:12:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T15:12:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:12:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T15:12:01.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:12:01.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:12:01.564 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:12:01.564 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:12:01.564 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:12:01.564 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:12:01.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:12:01.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:12:01.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:12:01.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:12:01.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:12:01.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:12:02.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:12:02.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:12:02.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:12:02.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-09-13T15:12:02.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:12:02.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-09-13T15:12:02.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-09-13T15:12:02.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-13T15:12:02.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:12:02.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-09-13T15:12:02.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-13T15:12:02.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:12:02.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:12:02.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:12:02.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:12:02.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-13T15:12:02.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:12:02.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:12:02.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:12:02.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:12:02.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:12:02.258 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:12:02.258 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:12:02.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:12:02.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:12:02.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:12:02.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:12:02.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-13T15:12:02.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:12:02.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:12:02.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:12:02.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:12:02.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:12:02.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-13T15:12:04.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:12:04.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:12:04.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:12:04.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:12:04.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:12:04.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:12:04.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:12:04.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:12:04.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:12:04.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:12:04.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:12:04.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:12:04.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:12:04.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:12:04.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:12:04.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:12:04.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:12:04.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:12:04.530 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:12:04.530 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:12:04.530 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:12:04.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:12:04.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:12:04.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:12:04.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:12:04.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:12:04.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:12:04.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:12:04.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:12:04.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:12:04.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:12:04.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:12:04.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:12:04.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:12:04.531 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:12:04.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:12:04.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:12:04.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:12:04.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:12:04.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:12:04.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:12:04.565 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:04.564+0000 7f3be5bad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:12:04.568 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:04.568+0000 7f3be5bad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:12:04.569 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:04.568+0000 7f3be5bad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:12:04.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:12:04.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:12:04.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:12:04.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:12:04.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:12:04.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:04.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:12:04.860 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:12:04.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:04.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:05.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:12:05.830 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:05.828+0000 7f3be5bad7c0 -1 Falling back to public interface 2024-09-13T15:12:06.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:12:06.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:06.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:12:06.199 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:12:06.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:06.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:06.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:12:07.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:12:07.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:07.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:12:07.537 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:12:07.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:07.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:07.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:12:08.628 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:08.627+0000 7f3be5bad7c0 -1 osd.1 97 log_to_monitors true 2024-09-13T15:12:08.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:12:08.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:08.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:12:08.877 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:12:08.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:08.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:09.203 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 101 up_thru 95 down_at 98 last_clean_interval [90,97) [v2:127.0.0.1:6810/1337243053,v1:127.0.0.1:6811/1337243053] [v2:127.0.0.1:6812/1337243053,v1:127.0.0.1:6813/1337243053] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:12:09.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:12:09.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:12:09.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:12:09.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:12:09.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:12:09.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:12:09.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:12:09.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:12:09.204 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:12:09.204 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:12:09.205 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:12:09.205 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:12:09.205 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:12:09.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:12:09.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:12:09.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:12:09.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:12:09.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:12:09.311 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:12:09.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:12:09.683 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:12:09.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:12:09.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:12:09.683 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:12:09.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=408021893125 2024-09-13T15:12:09.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 408021893125 2024-09-13T15:12:09.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-408021893125' 2024-09-13T15:12:09.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:12:09.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:12:10.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=433791696898 2024-09-13T15:12:10.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 433791696898 2024-09-13T15:12:10.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-408021893125 1-433791696898' 2024-09-13T15:12:10.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:12:10.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-408021893125 2024-09-13T15:12:10.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:12:10.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:12:10.005 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-408021893125 2024-09-13T15:12:10.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:12:10.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=408021893125 2024-09-13T15:12:10.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 408021893125' 2024-09-13T15:12:10.007 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 408021893125 2024-09-13T15:12:10.007 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:12:10.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893124 -lt 408021893125 2024-09-13T15:12:10.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:12:11.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:12:11.312 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:12:11.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893124 -lt 408021893125 2024-09-13T15:12:11.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:12:12.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T15:12:12.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:12:12.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893126 -lt 408021893125 2024-09-13T15:12:12.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:12:12.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-433791696898 2024-09-13T15:12:12.969 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:12:12.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:12:12.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-433791696898 2024-09-13T15:12:12.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:12:12.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=433791696898 2024-09-13T15:12:12.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 433791696898' 2024-09-13T15:12:12.974 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 433791696898 2024-09-13T15:12:12.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:12:13.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696898 -lt 433791696898 2024-09-13T15:12:13.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:12:13.289 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:12:13.289 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:12:13.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:12:13.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:12:13.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:12:13.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:12:13.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:12:13.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:12:13.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:12:13.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:12:14.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:12:14.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:12:14.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:12:14.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:12:14.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:12:14.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:12:14.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:12:14.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-09-13T15:12:14.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:12:14.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-09-13T15:12:14.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-09-13T15:12:14.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-13T15:12:14.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:12:14.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-09-13T15:12:14.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-09-13T15:12:14.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-09-13T15:12:14.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-13T15:12:14.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:12:14.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:12:14.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:12:14.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:12:14.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-13T15:12:14.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:12:14.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:12:14.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:12:14.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:12:14.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:12:14.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:12:14.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:12:14.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:12:14.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:12:14.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:12:14.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:12:14.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-13T15:12:14.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:12:14.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:12:14.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:12:14.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:12:14.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:12:14.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-13T15:12:16.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:12:16.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:12:16.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:12:16.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:12:16.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:12:16.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:12:16.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:12:16.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:12:16.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:12:16.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:12:16.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:12:16.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:12:16.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:12:16.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:12:16.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:12:16.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:12:16.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:12:16.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:12:16.845 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:12:16.845 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:12:16.846 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:12:16.846 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:12:16.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:12:16.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:12:16.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:12:16.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:12:16.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:12:16.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:12:16.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:12:16.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:12:16.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:12:16.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:12:16.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:12:16.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:12:16.847 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:12:16.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:12:16.847 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:12:16.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:12:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:12:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:12:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:12:16.882 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:16.879+0000 7f8d020db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:12:16.884 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:16.883+0000 7f8d020db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:12:16.887 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:16.883+0000 7f8d020db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:12:17.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:12:17.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:12:17.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:12:17.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:12:17.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:12:17.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:17.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:12:17.178 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:12:17.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:17.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:12:17.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:12:17.930 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:17.927+0000 7f8d020db7c0 -1 Falling back to public interface 2024-09-13T15:12:18.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:12:18.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:18.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:12:18.515 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:12:18.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:18.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:12:18.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:12:19.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:12:19.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:19.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:12:19.842 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:12:19.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:19.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:12:20.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:12:20.566 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:20.563+0000 7f8d020db7c0 -1 osd.0 102 log_to_monitors true 2024-09-13T15:12:21.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:12:21.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:21.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:12:21.173 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:12:21.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:21.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:12:21.606 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 106 up_thru 106 down_at 103 last_clean_interval [95,102) [v2:127.0.0.1:6802/920616811,v1:127.0.0.1:6803/920616811] [v2:127.0.0.1:6804/920616811,v1:127.0.0.1:6805/920616811] exists,up 2e31414c-0818-44ce-b14a-578b74fc94f5 2024-09-13T15:12:21.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:12:21.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:12:21.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:12:21.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:12:21.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:12:21.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:12:21.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:12:21.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:12:21.608 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:12:21.608 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:12:21.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:12:21.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:12:21.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:12:21.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:12:21.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:12:21.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:12:21.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:12:21.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:12:21.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:12:22.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:12:22.066 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:12:22.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:12:22.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:12:22.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:12:22.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533378 2024-09-13T15:12:22.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533378 2024-09-13T15:12:22.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533378' 2024-09-13T15:12:22.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:12:22.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:12:22.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=433791696901 2024-09-13T15:12:22.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 433791696901 2024-09-13T15:12:22.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533378 1-433791696901' 2024-09-13T15:12:22.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:12:22.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-455266533378 2024-09-13T15:12:22.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:12:22.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:12:22.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-455266533378 2024-09-13T15:12:22.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:12:22.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533378 2024-09-13T15:12:22.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 455266533378' 2024-09-13T15:12:22.362 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 455266533378 2024-09-13T15:12:22.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:12:22.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 455266533378 2024-09-13T15:12:22.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:12:23.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:12:23.678 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:12:24.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533378 -lt 455266533378 2024-09-13T15:12:24.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:12:24.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:12:24.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-433791696901 2024-09-13T15:12:24.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:12:24.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-433791696901 2024-09-13T15:12:24.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:12:24.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=433791696901 2024-09-13T15:12:24.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 433791696901' 2024-09-13T15:12:24.006 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 433791696901 2024-09-13T15:12:24.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:12:24.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696901 -lt 433791696901 2024-09-13T15:12:24.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:12:24.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:12:24.302 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:12:24.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:12:24.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:12:24.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:12:24.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:12:24.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:12:24.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:12:24.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:12:24.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:12:25.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:12:25.029 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:12:25.030 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:12:25.030 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:12:25.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:12:25.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:12:25.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:12:25.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-13T15:12:25.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:12:25.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:12:25.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:12:25.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:12:25.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-13T15:12:25.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:12:25.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:12:25.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:12:25.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:12:25.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:12:25.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:12:25.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:12:25.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:12:25.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:12:25.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:12:25.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:12:25.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-13T15:12:25.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:12:25.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:12:25.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:12:25.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:12:25.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:12:25.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-13T15:12:27.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:12:27.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:12:27.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:12:27.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:12:27.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:12:27.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:12:27.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:12:27.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:12:27.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:12:27.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:12:27.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:12:27.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:12:27.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:12:27.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:12:27.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:12:27.533 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:12:27.533 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:12:27.533 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:12:27.533 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:12:27.534 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:12:27.534 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:12:27.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:12:27.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:12:27.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:12:27.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:12:27.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:12:27.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:12:27.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:12:27.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:12:27.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:12:27.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:12:27.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:12:27.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:12:27.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:12:27.535 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:12:27.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:12:27.536 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:12:27.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:12:27.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:12:27.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:12:27.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:12:27.570 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:27.567+0000 7fb08454f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:12:27.573 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:27.571+0000 7fb08454f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:12:27.575 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:27.571+0000 7fb08454f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:12:27.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:12:27.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:12:27.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:12:27.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:12:27.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:12:27.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:27.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:12:27.880 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:12:27.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:27.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:28.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:12:29.070 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:29.067+0000 7fb08454f7c0 -1 Falling back to public interface 2024-09-13T15:12:29.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:12:29.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:29.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:12:29.228 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:12:29.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:29.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:29.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:12:30.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:12:30.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:30.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:12:30.567 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:12:30.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:30.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:30.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:12:31.832 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:31.831+0000 7fb08454f7c0 -1 osd.1 108 log_to_monitors true 2024-09-13T15:12:31.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:12:31.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:31.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:12:31.905 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:12:31.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:31.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:32.288 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 112 up_thru 107 down_at 109 last_clean_interval [101,108) [v2:127.0.0.1:6810/2017659801,v1:127.0.0.1:6811/2017659801] [v2:127.0.0.1:6812/2017659801,v1:127.0.0.1:6813/2017659801] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:12:32.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:12:32.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:12:32.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:12:32.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:12:32.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:12:32.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:12:32.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:12:32.289 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:12:32.289 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:12:32.290 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:12:32.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:12:32.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:12:32.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:12:32.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:12:32.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:12:32.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:12:32.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:12:32.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:12:32.390 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:12:32.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:12:32.768 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:12:32.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:12:32.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:12:32.769 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:12:33.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533381 2024-09-13T15:12:33.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533381 2024-09-13T15:12:33.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533381' 2024-09-13T15:12:33.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:12:33.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:12:33.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=481036337154 2024-09-13T15:12:33.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 481036337154 2024-09-13T15:12:33.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533381 1-481036337154' 2024-09-13T15:12:33.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:12:33.155 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-455266533381 2024-09-13T15:12:33.155 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:12:33.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:12:33.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-455266533381 2024-09-13T15:12:33.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:12:33.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533381 2024-09-13T15:12:33.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 455266533381' 2024-09-13T15:12:33.160 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 455266533381 2024-09-13T15:12:33.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:12:33.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533380 -lt 455266533381 2024-09-13T15:12:33.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:12:34.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:12:34.455 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:12:34.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533381 -lt 455266533381 2024-09-13T15:12:34.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:12:34.786 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-481036337154 2024-09-13T15:12:34.786 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:12:34.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:12:34.788 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-481036337154 2024-09-13T15:12:34.788 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:12:34.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=481036337154 2024-09-13T15:12:34.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 481036337154' 2024-09-13T15:12:34.789 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 481036337154 2024-09-13T15:12:34.789 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:12:35.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 481036337154 -lt 481036337154 2024-09-13T15:12:35.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:12:35.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:12:35.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:12:35.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:12:35.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:12:35.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:12:35.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:12:35.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:12:35.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:12:35.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:12:35.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:12:35.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:12:35.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:12:35.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:12:35.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:12:36.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:12:36.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:12:36.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:12:36.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-09-13T15:12:36.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-13T15:12:36.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:12:36.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:12:36.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:12:36.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:12:36.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-13T15:12:36.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:12:36.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:12:36.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:12:36.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:12:36.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:12:36.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:12:36.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:12:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:12:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:12:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:12:36.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:12:36.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-13T15:12:36.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:12:36.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:12:36.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:12:36.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:12:36.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:12:36.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-13T15:12:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:12:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:12:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:12:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:12:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:12:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:12:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:12:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:12:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:12:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:12:38.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:12:38.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:12:38.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:12:38.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:12:38.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:12:38.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:12:38.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:12:38.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:12:38.988 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:12:38.988 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:12:38.988 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:12:38.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:12:38.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:12:38.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:12:38.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:12:38.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:12:38.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:12:38.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:12:38.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:12:38.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:12:38.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:12:38.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:12:38.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:12:38.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:12:38.990 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:12:38.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:12:38.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:12:38.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:12:38.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:12:38.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:12:38.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:12:39.025 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:39.023+0000 7f9aba2987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:12:39.028 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:39.027+0000 7f9aba2987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:12:39.030 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:39.027+0000 7f9aba2987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:12:39.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:12:39.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:12:39.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:12:39.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:12:39.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:12:39.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:39.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:12:39.325 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:12:39.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:39.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:39.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:12:40.302 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:40.299+0000 7f9aba2987c0 -1 Falling back to public interface 2024-09-13T15:12:40.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:12:40.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:40.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:12:40.667 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:12:40.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:40.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:41.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:12:42.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:12:42.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:42.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:12:42.005 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:12:42.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:42.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:42.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:12:42.964 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:42.962+0000 7f9aba2987c0 -1 osd.1 113 log_to_monitors true 2024-09-13T15:12:43.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:12:43.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:43.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:12:43.345 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:12:43.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:43.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:43.872 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 117 up_thru 112 down_at 114 last_clean_interval [112,113) [v2:127.0.0.1:6810/3704034605,v1:127.0.0.1:6811/3704034605] [v2:127.0.0.1:6812/3704034605,v1:127.0.0.1:6813/3704034605] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:12:43.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:12:43.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:12:43.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:12:43.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:12:43.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:12:43.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:12:43.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:12:43.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:12:43.874 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:12:43.874 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:12:43.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:12:43.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:12:43.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:12:43.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:12:43.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:12:43.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:12:43.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:12:43.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:12:43.999 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:12:44.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:12:44.589 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:12:44.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:12:44.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:12:44.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:12:44.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533384 2024-09-13T15:12:44.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533384 2024-09-13T15:12:44.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533384' 2024-09-13T15:12:44.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:12:44.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:12:45.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=502511173634 2024-09-13T15:12:45.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 502511173634 2024-09-13T15:12:45.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533384 1-502511173634' 2024-09-13T15:12:45.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:12:45.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-455266533384 2024-09-13T15:12:45.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:12:45.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:12:45.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-455266533384 2024-09-13T15:12:45.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:12:45.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533384 2024-09-13T15:12:45.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 455266533384' 2024-09-13T15:12:45.006 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 455266533384 2024-09-13T15:12:45.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:12:45.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533383 -lt 455266533384 2024-09-13T15:12:45.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:12:46.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:12:46.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:12:46.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533384 -lt 455266533384 2024-09-13T15:12:46.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:12:46.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-502511173634 2024-09-13T15:12:46.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:12:46.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:12:46.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-502511173634 2024-09-13T15:12:46.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:12:46.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=502511173634 2024-09-13T15:12:46.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 502511173634' 2024-09-13T15:12:46.669 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 502511173634 2024-09-13T15:12:46.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:12:47.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 502511173634 -lt 502511173634 2024-09-13T15:12:47.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:12:47.016 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:12:47.016 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:12:47.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:12:47.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:12:47.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:12:47.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:12:47.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:12:47.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:12:47.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:12:47.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:12:47.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:12:47.732 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:12:47.732 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:12:47.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:12:48.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:12:48.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:12:48.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:12:48.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-09-13T15:12:48.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:12:48.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-09-13T15:12:48.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-09-13T15:12:48.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-13T15:12:48.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:12:48.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-09-13T15:12:48.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:12:48.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:12:48.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:12:48.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:12:48.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-09-13T15:12:48.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:12:48.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:12:48.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:12:48.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:12:48.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:12:48.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:12:48.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:12:48.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:12:48.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:12:48.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:12:48.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:12:48.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-09-13T15:12:48.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:12:48.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:12:48.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:12:48.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:12:48.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:12:48.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-09-13T15:12:50.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:12:50.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:12:50.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:12:50.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:12:50.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:12:50.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:12:50.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:12:50.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:12:50.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:12:50.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:12:50.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:12:50.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:12:50.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:12:50.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:12:50.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:12:50.555 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:12:50.555 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:12:50.555 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:12:50.555 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:12:50.555 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:12:50.555 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:12:50.555 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:12:50.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:12:50.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:12:50.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:12:50.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:12:50.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:12:50.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:12:50.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:12:50.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:12:50.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:12:50.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:12:50.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:12:50.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:12:50.557 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:12:50.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:12:50.557 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:12:50.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:12:50.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:12:50.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:12:50.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:12:50.584 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:50.582+0000 7f4c886fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:12:50.586 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:50.582+0000 7f4c886fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:12:50.587 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:50.586+0000 7f4c886fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:12:50.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:12:50.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:12:50.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:12:50.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:12:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:12:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:12:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:12:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:51.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:12:51.610 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:51.606+0000 7f4c886fc7c0 -1 Falling back to public interface 2024-09-13T15:12:52.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:12:52.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:52.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:12:52.235 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:12:52.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:52.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:52.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:12:53.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:12:53.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:53.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:12:53.556 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:12:53.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:53.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:53.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:12:54.355 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:12:54.350+0000 7f4c886fc7c0 -1 osd.1 119 log_to_monitors true 2024-09-13T15:12:54.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:12:54.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:12:54.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:12:54.901 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:12:54.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:12:54.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:12:55.279 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 123 up_thru 117 down_at 120 last_clean_interval [117,119) [v2:127.0.0.1:6810/3141932305,v1:127.0.0.1:6811/3141932305] [v2:127.0.0.1:6812/3141932305,v1:127.0.0.1:6813/3141932305] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:12:55.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:12:55.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:12:55.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:12:55.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:12:55.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:12:55.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:12:55.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:12:55.280 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:12:55.280 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:12:55.280 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:12:55.280 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:12:55.281 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:12:55.281 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:12:55.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:12:55.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:12:55.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:12:55.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:12:55.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:12:55.381 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:12:55.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:12:55.830 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:12:55.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:12:55.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:12:55.830 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:12:56.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533388 2024-09-13T15:12:56.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533388 2024-09-13T15:12:56.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533388' 2024-09-13T15:12:56.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:12:56.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:12:56.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=528280977410 2024-09-13T15:12:56.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 528280977410 2024-09-13T15:12:56.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533388 1-528280977410' 2024-09-13T15:12:56.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:12:56.306 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-455266533388 2024-09-13T15:12:56.306 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:12:56.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:12:56.307 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-455266533388 2024-09-13T15:12:56.307 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:12:56.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533388 2024-09-13T15:12:56.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 455266533388' 2024-09-13T15:12:56.307 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 455266533388 2024-09-13T15:12:56.308 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:12:56.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533386 -lt 455266533388 2024-09-13T15:12:56.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:12:57.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:12:57.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:12:57.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533388 -lt 455266533388 2024-09-13T15:12:57.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:12:57.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-528280977410 2024-09-13T15:12:57.984 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:12:57.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:12:57.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-528280977410 2024-09-13T15:12:57.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:12:57.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=528280977410 2024-09-13T15:12:57.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 528280977410' 2024-09-13T15:12:57.989 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 528280977410 2024-09-13T15:12:57.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:12:58.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 528280977410 -lt 528280977410 2024-09-13T15:12:58.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:12:58.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:12:58.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:12:58.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:12:58.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:12:58.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:12:58.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:12:58.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:12:58.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:12:58.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:12:58.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:12:59.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:12:59.075 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:12:59.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:12:59.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:12:59.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:12:59.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:12:59.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:12:59.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-09-13T15:12:59.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-09-13T15:12:59.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-13T15:12:59.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:12:59.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:12:59.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:12:59.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:12:59.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-13T15:12:59.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:12:59.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:12:59.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:12:59.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:12:59.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:12:59.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:12:59.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:12:59.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:12:59.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:12:59.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:12:59.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:12:59.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-13T15:12:59.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:12:59.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:12:59.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:12:59.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:12:59.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:12:59.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-13T15:13:02.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:13:02.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:13:02.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:13:02.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:13:02.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:13:02.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:13:02.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:13:02.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:13:02.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:13:02.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:13:02.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:13:02.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:13:02.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:13:02.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:13:02.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:13:02.190 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:13:02.190 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:13:02.190 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:13:02.190 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:13:02.190 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:13:02.190 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:13:02.190 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:13:02.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:13:02.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:13:02.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:13:02.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:13:02.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:13:02.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:13:02.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:13:02.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:13:02.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:13:02.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:13:02.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:13:02.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:13:02.191 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:13:02.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:13:02.192 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:13:02.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:13:02.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:13:02.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:13:02.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:13:02.226 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:02.222+0000 7f6cedca57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:13:02.229 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:02.226+0000 7f6cedca57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:13:02.231 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:02.226+0000 7f6cedca57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:13:02.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:13:02.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:13:02.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:13:02.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:13:02.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:13:02.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:02.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:13:02.522 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:13:02.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:02.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:13:02.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:03.726 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:03.722+0000 7f6cedca57c0 -1 Falling back to public interface 2024-09-13T15:13:03.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:03.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:03.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:13:03.874 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:13:03.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:03.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:13:04.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:05.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:05.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:05.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:13:05.211 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:13:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:13:05.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:06.315 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:06.310+0000 7f6cedca57c0 -1 osd.1 124 log_to_monitors true 2024-09-13T15:13:06.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:06.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:06.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:13:06.545 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:13:06.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:06.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:13:06.907 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 128 up_thru 123 down_at 125 last_clean_interval [123,124) [v2:127.0.0.1:6810/4137488367,v1:127.0.0.1:6811/4137488367] [v2:127.0.0.1:6812/4137488367,v1:127.0.0.1:6813/4137488367] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:13:06.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:13:06.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:13:06.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:13:06.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:13:06.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:13:06.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:13:06.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:13:06.908 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:13:06.908 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:13:06.909 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:13:06.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:13:06.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:13:06.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:13:07.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:13:07.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:13:07.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:13:07.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:13:07.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:13:07.033 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:13:07.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:13:07.411 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:13:07.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:13:07.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:13:07.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:13:07.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533391 2024-09-13T15:13:07.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533391 2024-09-13T15:13:07.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533391' 2024-09-13T15:13:07.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:13:07.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:13:07.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=549755813890 2024-09-13T15:13:07.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 549755813890 2024-09-13T15:13:07.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533391 1-549755813890' 2024-09-13T15:13:07.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:13:07.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-455266533391 2024-09-13T15:13:07.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:13:07.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:13:07.856 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-455266533391 2024-09-13T15:13:07.856 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:13:07.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533391 2024-09-13T15:13:07.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 455266533391' 2024-09-13T15:13:07.857 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 455266533391 2024-09-13T15:13:07.858 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:13:08.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533390 -lt 455266533391 2024-09-13T15:13:08.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:13:09.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:13:09.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:13:09.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533390 -lt 455266533391 2024-09-13T15:13:09.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:13:10.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T15:13:10.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:13:10.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533391 -lt 455266533391 2024-09-13T15:13:10.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:13:10.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:13:10.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-549755813890 2024-09-13T15:13:10.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:13:10.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:13:10.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-549755813890 2024-09-13T15:13:10.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=549755813890 2024-09-13T15:13:10.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 549755813890' 2024-09-13T15:13:10.992 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 549755813890 2024-09-13T15:13:10.993 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:13:11.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 549755813890 -lt 549755813890 2024-09-13T15:13:11.311 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:13:11.312 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:13:11.312 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:13:11.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:13:11.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:13:11.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:13:11.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:13:11.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:13:11.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:13:11.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:13:11.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:13:12.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:13:12.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:13:12.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:13:12.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:13:12.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:13:12.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:13:12.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:13:12.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-09-13T15:13:12.440 INFO:tasks.workunit.client.0.smithi112.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-09-13T15:13:12.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:13:12.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-09-13T15:13:12.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-09-13T15:13:12.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-13T15:13:12.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:13:12.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:13:12.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-09-13T15:13:12.443 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-09-13T15:13:12.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-13T15:13:12.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:13:12.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:13:12.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-09-13T15:13:12.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-09-13T15:13:12.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-13T15:13:12.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:13:12.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:13:12.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-09-13T15:13:12.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-09-13T15:13:12.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-13T15:13:12.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:13:12.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:13:12.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-09-13T15:13:12.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-09-13T15:13:12.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-13T15:13:12.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:13:12.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-09-13T15:13:12.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-13T15:13:12.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:13:12.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:13:12.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:13:12.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:13:12.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-13T15:13:12.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:13:12.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:13:12.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:13:12.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:13:12.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:13:12.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:13:12.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:13:12.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:13:12.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:13:12.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:13:12.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:13:12.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-13T15:13:12.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:13:12.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:13:12.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:13:12.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:13:12.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:13:12.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-13T15:13:14.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:13:14.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:13:14.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:13:14.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:13:14.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:13:14.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:13:14.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:13:14.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:13:14.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:13:14.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:13:14.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:13:14.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:13:14.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:13:14.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:13:14.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:13:14.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:13:14.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:13:14.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:13:14.842 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:13:14.843 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:13:14.843 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:13:14.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:13:14.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:13:14.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:13:14.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:13:14.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:13:14.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:13:14.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:13:14.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:13:14.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:13:14.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:13:14.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:13:14.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:13:14.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:13:14.844 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:13:14.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:13:14.847 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:13:14.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:13:14.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:13:14.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:13:14.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:13:14.878 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:14.874+0000 7f39ab4b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:13:14.881 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:14.878+0000 7f39ab4b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:13:14.883 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:14.878+0000 7f39ab4b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:13:15.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:13:15.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:13:15.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:13:15.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:13:15.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:13:15.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:15.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:13:15.176 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:13:15.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:13:15.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:15.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:16.368 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:16.361+0000 7f39ab4b17c0 -1 Falling back to public interface 2024-09-13T15:13:16.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:16.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:16.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:13:16.512 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:13:16.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:16.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:13:16.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:17.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:17.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:17.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:13:17.830 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:13:17.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:17.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:13:18.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:19.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:19.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:19.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:13:19.155 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:13:19.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:19.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:13:19.192 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:19.189+0000 7f39ab4b17c0 -1 osd.0 130 log_to_monitors true 2024-09-13T15:13:19.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:20.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:20.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:20.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:13:20.505 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:13:20.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:20.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:13:20.842 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 134 up_thru 134 down_at 131 last_clean_interval [106,130) [v2:127.0.0.1:6802/3357275439,v1:127.0.0.1:6803/3357275439] [v2:127.0.0.1:6804/3357275439,v1:127.0.0.1:6805/3357275439] exists,up 2e31414c-0818-44ce-b14a-578b74fc94f5 2024-09-13T15:13:20.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:13:20.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:13:20.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:13:20.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:13:20.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:13:20.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:13:20.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:13:20.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:13:20.843 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:13:20.843 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:13:20.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:13:20.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:13:20.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:13:20.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:13:20.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:13:20.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:13:20.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:13:20.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:13:20.955 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:13:21.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:13:21.286 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:13:21.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:13:21.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:13:21.286 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:13:21.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=575525617666 2024-09-13T15:13:21.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 575525617666 2024-09-13T15:13:21.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-575525617666' 2024-09-13T15:13:21.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:13:21.461 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:13:21.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=549755813894 2024-09-13T15:13:21.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 549755813894 2024-09-13T15:13:21.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-575525617666 1-549755813894' 2024-09-13T15:13:21.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:13:21.603 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-575525617666 2024-09-13T15:13:21.603 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:13:21.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:13:21.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-575525617666 2024-09-13T15:13:21.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:13:21.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=575525617666 2024-09-13T15:13:21.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 575525617666' 2024-09-13T15:13:21.607 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 575525617666 2024-09-13T15:13:21.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:13:21.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 575525617666 -lt 575525617666 2024-09-13T15:13:21.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:13:21.952 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-549755813894 2024-09-13T15:13:21.952 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:13:21.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:13:21.954 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-549755813894 2024-09-13T15:13:21.954 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:13:21.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=549755813894 2024-09-13T15:13:21.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 549755813894' 2024-09-13T15:13:21.956 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 549755813894 2024-09-13T15:13:21.956 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:13:22.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 549755813893 -lt 549755813894 2024-09-13T15:13:22.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:13:23.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:13:23.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:13:23.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 549755813893 -lt 549755813894 2024-09-13T15:13:23.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:13:24.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T15:13:24.613 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:13:24.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 549755813894 -lt 549755813894 2024-09-13T15:13:24.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:13:24.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:13:24.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:13:25.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:13:25.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:13:25.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:13:25.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:13:25.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:13:25.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:13:25.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:13:25.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:13:25.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:13:25.676 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:13:25.676 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:13:25.676 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:13:26.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:13:26.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:13:26.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:13:26.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-09-13T15:13:26.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:13:26.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:13:26.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:13:26.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:13:26.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-09-13T15:13:26.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:13:26.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:13:26.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:13:26.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:13:26.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:13:26.050 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:13:26.050 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:13:26.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:13:26.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:13:26.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:13:26.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:13:26.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-09-13T15:13:26.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:13:26.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:13:26.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:13:26.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:13:26.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:13:26.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-09-13T15:13:28.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:13:28.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:13:28.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:13:28.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:13:28.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:13:28.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:13:28.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:13:28.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:13:28.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:13:28.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:13:28.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:13:28.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:13:28.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:13:28.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:13:28.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:13:28.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:13:28.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:13:28.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:13:28.513 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:13:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:13:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:13:28.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:13:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:13:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:13:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:13:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:13:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:13:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:13:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:13:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:13:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:13:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:13:28.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:13:28.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:13:28.515 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:13:28.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:13:28.516 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:13:28.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:13:28.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:13:28.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:13:28.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:13:28.550 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:28.545+0000 7f504ce117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:13:28.552 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:28.549+0000 7f504ce117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:13:28.554 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:28.549+0000 7f504ce117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:13:28.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:13:28.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:13:28.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:13:28.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:13:28.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:13:28.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:28.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:13:28.851 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:13:28.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:28.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:13:29.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:29.791 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:29.785+0000 7f504ce117c0 -1 Falling back to public interface 2024-09-13T15:13:30.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:30.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:30.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:13:30.202 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:13:30.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:30.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:13:30.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:31.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:31.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:31.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:13:31.523 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:13:31.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:31.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:13:31.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:32.467 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:32.461+0000 7f504ce117c0 -1 osd.1 135 log_to_monitors true 2024-09-13T15:13:32.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:32.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:32.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:13:32.855 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:13:32.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:32.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:13:33.190 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 139 up_thru 134 down_at 136 last_clean_interval [128,135) [v2:127.0.0.1:6810/3676855312,v1:127.0.0.1:6811/3676855312] [v2:127.0.0.1:6812/3676855312,v1:127.0.0.1:6813/3676855312] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:13:33.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:13:33.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:13:33.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:13:33.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:13:33.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:13:33.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:13:33.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:13:33.191 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:13:33.192 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:13:33.192 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:13:33.192 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:13:33.192 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:13:33.192 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:13:33.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:13:33.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:13:33.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:13:33.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:13:33.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:13:33.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:13:33.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:13:33.786 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:13:33.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:13:33.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:13:33.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:13:33.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=575525617669 2024-09-13T15:13:33.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 575525617669 2024-09-13T15:13:33.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-575525617669' 2024-09-13T15:13:33.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:13:33.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:13:34.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=597000454146 2024-09-13T15:13:34.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 597000454146 2024-09-13T15:13:34.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-575525617669 1-597000454146' 2024-09-13T15:13:34.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:13:34.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-575525617669 2024-09-13T15:13:34.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:13:34.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:13:34.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-575525617669 2024-09-13T15:13:34.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:13:34.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=575525617669 2024-09-13T15:13:34.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 575525617669' 2024-09-13T15:13:34.108 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 575525617669 2024-09-13T15:13:34.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:13:34.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 575525617668 -lt 575525617669 2024-09-13T15:13:34.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:13:35.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:13:35.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:13:35.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 575525617670 -lt 575525617669 2024-09-13T15:13:35.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:13:35.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-597000454146 2024-09-13T15:13:35.734 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:13:35.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:13:35.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-597000454146 2024-09-13T15:13:35.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:13:35.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=597000454146 2024-09-13T15:13:35.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 597000454146' 2024-09-13T15:13:35.738 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 597000454146 2024-09-13T15:13:35.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:13:36.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 597000454146 -lt 597000454146 2024-09-13T15:13:36.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:13:36.080 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:13:36.080 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:13:36.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:13:36.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:13:36.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:13:36.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:13:36.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:13:36.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:13:36.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:13:36.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:13:36.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:13:36.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:13:36.828 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:13:36.828 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:13:37.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:13:37.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:13:37.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:13:37.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-09-13T15:13:37.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:13:37.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-09-13T15:13:37.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-09-13T15:13:37.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-13T15:13:37.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:13:37.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-09-13T15:13:37.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:13:37.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:13:37.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:13:37.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:13:37.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-09-13T15:13:37.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:13:37.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:13:37.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:13:37.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:13:37.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:13:37.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:13:37.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:13:37.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:13:37.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:13:37.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:13:37.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:13:37.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-09-13T15:13:37.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:13:37.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:13:37.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:13:37.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:13:37.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:13:37.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-09-13T15:13:38.893 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-13T15:13:39.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:13:39.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:13:39.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:13:39.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:13:39.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:13:39.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:13:39.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:13:39.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:13:39.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:13:39.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:13:39.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:13:39.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:13:39.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:13:39.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:13:39.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:13:39.433 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:13:39.433 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:13:39.433 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:13:39.433 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:13:39.434 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:13:39.434 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:13:39.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:13:39.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:13:39.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:13:39.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:13:39.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:13:39.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:13:39.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:13:39.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:13:39.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:13:39.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:13:39.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:13:39.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:13:39.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:13:39.436 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:13:39.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:13:39.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:13:39.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:13:39.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:13:39.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:13:39.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:13:39.475 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:39.469+0000 7f0e6153a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:13:39.478 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:39.473+0000 7f0e6153a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:13:39.479 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:39.473+0000 7f0e6153a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:13:39.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:13:39.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:13:39.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:13:39.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:13:39.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:13:39.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:39.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:13:39.808 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:13:39.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:39.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:13:40.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:40.534 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:40.529+0000 7f0e6153a7c0 -1 Falling back to public interface 2024-09-13T15:13:41.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:41.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:41.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:13:41.179 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:13:41.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:41.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:13:41.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:42.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:42.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:42.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:13:42.513 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:13:42.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:42.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:13:42.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:43.808 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:43.805+0000 7f0e6153a7c0 -1 osd.1 140 log_to_monitors true 2024-09-13T15:13:43.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:43.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:43.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:13:43.846 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:13:43.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:43.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:13:44.232 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 144 up_thru 139 down_at 141 last_clean_interval [139,140) [v2:127.0.0.1:6810/2829100732,v1:127.0.0.1:6811/2829100732] [v2:127.0.0.1:6812/2829100732,v1:127.0.0.1:6813/2829100732] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:13:44.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:13:44.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:13:44.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:13:44.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:13:44.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:13:44.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:13:44.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:13:44.233 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:13:44.233 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:13:44.233 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:13:44.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:13:44.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:13:44.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:13:44.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:13:44.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:13:44.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:13:44.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:13:44.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:13:44.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:13:44.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:13:44.691 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:13:44.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:13:44.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:13:44.691 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:13:44.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=575525617673 2024-09-13T15:13:44.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 575525617673 2024-09-13T15:13:44.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-575525617673' 2024-09-13T15:13:44.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:13:44.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:13:45.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=618475290626 2024-09-13T15:13:45.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 618475290626 2024-09-13T15:13:45.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-575525617673 1-618475290626' 2024-09-13T15:13:45.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:13:45.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-575525617673 2024-09-13T15:13:45.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:13:45.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:13:45.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:13:45.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-575525617673 2024-09-13T15:13:45.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=575525617673 2024-09-13T15:13:45.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 575525617673' 2024-09-13T15:13:45.017 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 575525617673 2024-09-13T15:13:45.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:13:45.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 575525617671 -lt 575525617673 2024-09-13T15:13:45.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:13:46.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:13:46.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:13:46.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 575525617673 -lt 575525617673 2024-09-13T15:13:46.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:13:46.678 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-618475290626 2024-09-13T15:13:46.678 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:13:46.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:13:46.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-618475290626 2024-09-13T15:13:46.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:13:46.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=618475290626 2024-09-13T15:13:46.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 618475290626' 2024-09-13T15:13:46.682 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 618475290626 2024-09-13T15:13:46.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:13:47.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 618475290626 -lt 618475290626 2024-09-13T15:13:47.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:13:47.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:13:47.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:13:47.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:13:47.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:13:47.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:13:47.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:13:47.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:13:47.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:13:47.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:13:47.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:13:47.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:13:47.753 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:13:47.754 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:13:47.754 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:13:48.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-13T15:13:48.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-13T15:13:48.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:13:48.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-13T15:13:48.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:13:48.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:13:48.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:13:48.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:13:48.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:13:48.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:13:48.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:13:48.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:13:48.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:13:48.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:13:48.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:13:48.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:13:48.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:13:48.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:13:48.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-13T15:13:48.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-13T15:13:48.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:13:48.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:13:48.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:13:48.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:13:48.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:13:48.965 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:13:48.965 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:13:49.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:13:49.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:13:49.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T15:13:49.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:13:49.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T15:13:49.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:13:49.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:13:49.553 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:13:49.553 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:13:49.554 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:13:49.554 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:13:49.554 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:13:49.554 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:13:49.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:13:49.880 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:13:49.880 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:13:49.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:13:50.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:13:50.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:13:50.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:13:50.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:13:50.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-09-13T15:13:50.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-09-13T15:13:50.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-13T15:13:50.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:13:50.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-09-13T15:13:50.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:13:50.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:13:50.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:13:50.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:13:50.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-09-13T15:13:50.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:13:50.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:13:50.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:13:50.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:13:50.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:13:50.276 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:13:50.276 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:13:50.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:13:50.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:13:50.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:13:50.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:13:50.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-09-13T15:13:50.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:13:50.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:13:50.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:13:50.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:13:50.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:13:50.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-09-13T15:13:52.198 INFO:tasks.workunit.client.0.smithi112.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-13T15:13:52.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:13:52.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:13:52.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:13:52.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:13:52.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:13:52.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:13:52.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:13:52.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:13:52.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:13:52.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:13:52.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:13:52.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:13:52.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:13:52.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:13:52.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:13:52.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:13:52.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:13:52.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:13:52.990 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:13:52.990 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:13:52.990 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:13:52.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:13:52.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:13:52.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:13:52.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:13:52.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:13:52.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:13:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:13:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:13:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:13:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:13:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:13:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:13:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:13:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:13:52.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:13:52.992 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:13:52.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:13:52.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:13:52.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:13:52.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:13:53.026 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:53.020+0000 7f61a599d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:13:53.029 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:53.024+0000 7f61a599d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:13:53.030 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:53.024+0000 7f61a599d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:13:53.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:13:53.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:13:53.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:13:53.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:13:53.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:13:53.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:53.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:13:53.329 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:13:53.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:53.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:13:53.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:54.342 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:54.336+0000 7f61a599d7c0 -1 Falling back to public interface 2024-09-13T15:13:54.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:54.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:54.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:13:54.668 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:13:54.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:54.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:13:55.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:56.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:56.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:56.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:13:56.003 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:13:56.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:56.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:13:56.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:13:57.010 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:13:57.004+0000 7f61a599d7c0 -1 osd.0 145 log_to_monitors true 2024-09-13T15:13:57.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:13:57.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:13:57.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:13:57.325 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:13:57.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:13:57.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:13:57.664 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 149 up_thru 144 down_at 146 last_clean_interval [134,145) [v2:127.0.0.1:6802/700632816,v1:127.0.0.1:6803/700632816] [v2:127.0.0.1:6804/700632816,v1:127.0.0.1:6805/700632816] exists,up 2e31414c-0818-44ce-b14a-578b74fc94f5 2024-09-13T15:13:57.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:13:57.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:13:57.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:13:57.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:13:57.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:13:57.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:13:57.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:13:57.666 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:13:57.666 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:13:57.666 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:13:57.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:13:57.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:13:57.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:13:57.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:13:57.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:13:57.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:13:57.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:13:57.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:13:57.786 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:13:58.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:13:58.152 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:13:58.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:13:58.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:13:58.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:13:58.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=639950127106 2024-09-13T15:13:58.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 639950127106 2024-09-13T15:13:58.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-639950127106' 2024-09-13T15:13:58.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:13:58.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:13:58.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=618475290629 2024-09-13T15:13:58.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 618475290629 2024-09-13T15:13:58.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-639950127106 1-618475290629' 2024-09-13T15:13:58.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:13:58.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-639950127106 2024-09-13T15:13:58.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:13:58.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:13:58.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-639950127106 2024-09-13T15:13:58.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:13:58.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=639950127106 2024-09-13T15:13:58.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2024-09-13T15:13:58.527 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 639950127106 2024-09-13T15:13:58.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:13:58.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 639950127106 2024-09-13T15:13:58.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:13:59.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:13:59.836 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:14:00.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 639950127106 -lt 639950127106 2024-09-13T15:14:00.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:14:00.163 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-618475290629 2024-09-13T15:14:00.163 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:14:00.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:14:00.165 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-618475290629 2024-09-13T15:14:00.165 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:14:00.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=618475290629 2024-09-13T15:14:00.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 618475290629' 2024-09-13T15:14:00.167 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 618475290629 2024-09-13T15:14:00.167 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:14:00.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 618475290630 -lt 618475290629 2024-09-13T15:14:00.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:14:00.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:14:00.492 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:14:00.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:14:00.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:14:00.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:14:00.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:14:00.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:14:00.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:14:00.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:14:00.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:14:01.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:14:01.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:14:01.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:14:01.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:14:01.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:14:01.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:14:01.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:14:01.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-09-13T15:14:01.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-13T15:14:01.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:14:01.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:14:01.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:14:01.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:14:01.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-13T15:14:01.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:14:01.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:14:01.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:14:01.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:14:01.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:14:01.626 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:14:01.626 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:14:01.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:14:01.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:14:01.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:14:01.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:14:01.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-13T15:14:01.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:14:01.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:14:01.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:14:01.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:14:01.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:14:01.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-13T15:14:03.355 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-13T15:14:03.355 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-13T15:14:03.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:14:03.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:14:03.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:14:03.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:14:03.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:14:03.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:14:03.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:14:03.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:14:03.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:14:03.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:14:03.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:14:03.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:14:03.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:14:03.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:14:03.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:14:03.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:14:03.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:14:03.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:14:03.901 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:14:03.901 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:14:03.901 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:14:03.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:14:03.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:14:03.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:14:03.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:14:03.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:14:03.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:14:03.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:14:03.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:14:03.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:14:03.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:14:03.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:14:03.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:14:03.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:14:03.904 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:14:03.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:14:03.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:14:03.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:14:03.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:14:03.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:14:03.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:14:03.939 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:03.932+0000 7f09c4d867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:14:03.951 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:03.944+0000 7f09c4d867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:14:03.953 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:03.948+0000 7f09c4d867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:14:04.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:14:04.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:14:04.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:14:04.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:14:04.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:14:04.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:04.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:14:04.238 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:14:04.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:04.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:14:04.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:05.231 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:05.228+0000 7f09c4d867c0 -1 Falling back to public interface 2024-09-13T15:14:05.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:05.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:05.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:14:05.584 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:14:05.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:05.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:14:05.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:06.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:06.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:06.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:14:06.966 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:14:06.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:06.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:14:07.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:08.135 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:08.132+0000 7f09c4d867c0 -1 osd.1 150 log_to_monitors true 2024-09-13T15:14:08.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:08.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:08.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:14:08.305 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:14:08.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:08.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:14:08.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:09.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:09.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:09.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:14:09.644 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:14:09.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:09.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:14:09.992 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [144,150) [v2:127.0.0.1:6810/2988686338,v1:127.0.0.1:6811/2988686338] [v2:127.0.0.1:6812/2988686338,v1:127.0.0.1:6813/2988686338] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:14:09.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:14:09.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:14:09.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:14:09.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:14:09.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:14:09.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:14:09.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:14:09.993 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:14:09.994 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:14:09.994 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:14:09.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:14:09.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:14:09.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:14:10.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:14:10.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:14:10.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:14:10.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:14:10.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:14:10.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:14:10.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:14:10.441 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:14:10.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:14:10.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:14:10.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:14:10.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=639950127109 2024-09-13T15:14:10.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 639950127109 2024-09-13T15:14:10.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-639950127109' 2024-09-13T15:14:10.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:14:10.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:14:10.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=661424963586 2024-09-13T15:14:10.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 661424963586 2024-09-13T15:14:10.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-639950127109 1-661424963586' 2024-09-13T15:14:10.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:14:10.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-639950127109 2024-09-13T15:14:10.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:14:10.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:14:10.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-639950127109 2024-09-13T15:14:10.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:14:10.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=639950127109 2024-09-13T15:14:10.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 639950127109' 2024-09-13T15:14:10.742 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 639950127109 2024-09-13T15:14:10.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:14:11.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 639950127108 -lt 639950127109 2024-09-13T15:14:11.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:14:12.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:14:12.038 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:14:12.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 639950127109 -lt 639950127109 2024-09-13T15:14:12.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:14:12.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-661424963586 2024-09-13T15:14:12.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:14:12.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:14:12.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-661424963586 2024-09-13T15:14:12.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:14:12.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=661424963586 2024-09-13T15:14:12.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2024-09-13T15:14:12.359 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 661424963586 2024-09-13T15:14:12.359 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:14:12.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 661424963586 -lt 661424963586 2024-09-13T15:14:12.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:14:12.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:14:12.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:14:13.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:14:13.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:14:13.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:14:13.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:14:13.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:14:13.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:14:13.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:14:13.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:14:13.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:14:13.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:14:13.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:14:13.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:14:13.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:14:13.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:14:13.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:14:13.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:14:13.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-09-13T15:14:13.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-09-13T15:14:13.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-13T15:14:13.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:14:13.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-09-13T15:14:13.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-09-13T15:14:13.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-13T15:14:13.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:14:13.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:14:13.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:14:13.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:14:13.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-13T15:14:13.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:14:13.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:14:13.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:14:13.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:14:13.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:14:13.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:14:13.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:14:13.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:14:13.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:14:13.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:14:14.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:14:14.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-13T15:14:14.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:14:14.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:14:14.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:14:14.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:14:14.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:14:14.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-13T15:14:15.479 INFO:tasks.workunit.client.0.smithi112.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-13T15:14:16.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:14:16.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:14:16.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:14:16.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:14:16.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:14:16.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:14:16.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:14:16.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:14:16.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:14:16.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:14:16.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:14:16.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:14:16.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:14:16.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:14:16.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:14:16.345 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:14:16.345 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:14:16.345 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:14:16.345 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:14:16.345 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:14:16.346 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:14:16.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:14:16.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:14:16.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:14:16.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:14:16.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:14:16.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:14:16.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:14:16.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:14:16.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:14:16.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:14:16.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:14:16.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:14:16.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:14:16.348 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:14:16.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:14:16.349 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:14:16.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:14:16.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:14:16.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:14:16.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:14:16.383 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:16.380+0000 7ff927c747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:14:16.385 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:16.384+0000 7ff927c747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:14:16.386 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:16.384+0000 7ff927c747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:14:16.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:14:16.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:14:16.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:14:16.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:14:16.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:14:16.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:16.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:14:16.689 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:14:16.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:16.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:14:17.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:17.451 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:17.448+0000 7ff927c747c0 -1 Falling back to public interface 2024-09-13T15:14:18.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:18.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:18.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:14:18.030 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:14:18.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:18.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:14:18.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:19.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:19.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:19.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:14:19.345 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:14:19.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:19.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:14:19.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:20.286 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:20.284+0000 7ff927c747c0 -1 osd.0 156 log_to_monitors true 2024-09-13T15:14:20.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:20.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:20.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:14:20.697 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:14:20.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:20.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:14:21.177 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 160 up_thru 154 down_at 157 last_clean_interval [149,156) [v2:127.0.0.1:6802/1351268212,v1:127.0.0.1:6803/1351268212] [v2:127.0.0.1:6804/1351268212,v1:127.0.0.1:6805/1351268212] exists,up 2e31414c-0818-44ce-b14a-578b74fc94f5 2024-09-13T15:14:21.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:14:21.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:14:21.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:14:21.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:14:21.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:14:21.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:14:21.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:14:21.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:14:21.178 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:14:21.178 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:14:21.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:14:21.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:14:21.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:14:21.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:14:21.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:14:21.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:14:21.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:14:21.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:14:21.296 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:14:21.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:14:21.630 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:14:21.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:14:21.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:14:21.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:14:21.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=687194767362 2024-09-13T15:14:21.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 687194767362 2024-09-13T15:14:21.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-687194767362' 2024-09-13T15:14:21.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:14:21.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:14:21.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=661424963589 2024-09-13T15:14:21.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 661424963589 2024-09-13T15:14:21.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-687194767362 1-661424963589' 2024-09-13T15:14:21.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:14:21.947 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-687194767362 2024-09-13T15:14:21.947 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:14:21.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:14:21.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-687194767362 2024-09-13T15:14:21.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:14:21.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=687194767362 2024-09-13T15:14:21.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 687194767362' 2024-09-13T15:14:21.951 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 687194767362 2024-09-13T15:14:21.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:14:22.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 687194767362 2024-09-13T15:14:22.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:14:23.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:14:23.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:14:23.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 687194767362 2024-09-13T15:14:23.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:14:24.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T15:14:24.587 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:14:24.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 687194767362 -lt 687194767362 2024-09-13T15:14:24.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:14:24.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-661424963589 2024-09-13T15:14:24.906 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:14:24.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:14:24.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-661424963589 2024-09-13T15:14:24.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:14:24.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=661424963589 2024-09-13T15:14:24.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 661424963589' 2024-09-13T15:14:24.910 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 661424963589 2024-09-13T15:14:24.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:14:25.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 661424963590 -lt 661424963589 2024-09-13T15:14:25.243 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:14:25.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:14:25.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:14:25.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:14:25.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:14:25.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:14:25.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:14:25.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:14:25.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:14:25.636 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:14:25.636 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:14:25.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:14:25.998 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:14:25.998 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:14:25.999 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:14:26.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-13T15:14:26.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-13T15:14:26.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:14:26.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-13T15:14:26.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:14:26.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:14:26.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:14:26.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:14:26.477 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:14:26.477 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:14:26.477 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:14:26.477 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:14:26.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:14:26.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:14:26.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:14:26.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:14:26.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:14:26.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:14:27.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-13T15:14:27.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-13T15:14:27.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:14:27.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:14:27.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:14:27.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:14:27.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:14:27.219 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:14:27.220 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:14:27.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:14:27.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:14:27.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T15:14:27.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:14:27.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T15:14:27.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:14:27.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:14:27.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:14:27.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:14:27.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:14:27.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:14:27.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:14:27.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:14:28.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:14:28.122 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:14:28.122 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:14:28.123 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:14:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:14:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:14:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:14:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-13T15:14:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:14:28.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:14:28.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:14:28.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:14:28.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-13T15:14:28.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:14:28.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:14:28.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:14:28.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:14:28.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:14:28.515 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:14:28.515 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:14:28.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:14:28.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:14:28.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:14:28.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:14:28.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-13T15:14:28.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:14:28.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:14:28.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:14:28.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:14:28.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:14:28.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-13T15:14:30.032 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-13T15:14:30.033 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-13T15:14:30.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:14:30.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:14:30.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:14:30.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:14:30.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:14:30.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:14:30.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:14:30.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:14:30.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:14:30.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:14:30.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:14:30.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:14:30.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:14:30.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:14:30.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:14:30.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:14:30.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:14:30.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:14:30.614 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:14:30.615 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:14:30.615 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:14:30.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:14:30.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:14:30.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:14:30.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:14:30.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:14:30.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:14:30.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:14:30.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:14:30.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:14:30.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:14:30.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:14:30.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:14:30.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:14:30.616 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:14:30.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:14:30.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:14:30.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:14:30.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:14:30.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:14:30.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:14:30.651 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:30.647+0000 7efdd6a257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:14:30.653 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:30.651+0000 7efdd6a257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:14:30.655 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:30.651+0000 7efdd6a257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:14:30.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:14:30.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:14:30.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:14:30.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:14:30.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:14:30.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:30.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:14:30.946 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:14:30.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:30.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:14:31.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:31.719 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:31.715+0000 7efdd6a257c0 -1 Falling back to public interface 2024-09-13T15:14:32.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:32.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:32.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:14:32.297 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:14:32.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:32.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:14:32.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:33.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:33.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:33.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:14:33.633 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:14:33.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:33.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:14:33.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:34.194 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:34.191+0000 7efdd6a257c0 -1 osd.1 162 log_to_monitors true 2024-09-13T15:14:34.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:34.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:34.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:14:34.989 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:14:34.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:34.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:14:35.506 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 166 up_thru 166 down_at 163 last_clean_interval [154,162) [v2:127.0.0.1:6810/1353857217,v1:127.0.0.1:6811/1353857217] [v2:127.0.0.1:6812/1353857217,v1:127.0.0.1:6813/1353857217] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:14:35.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:14:35.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:14:35.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:14:35.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:14:35.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:14:35.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:14:35.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:14:35.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:14:35.508 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:14:35.508 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:14:35.508 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:14:35.508 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:14:35.508 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:14:35.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:14:35.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:14:35.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:14:35.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:14:35.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:14:35.628 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:14:36.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:14:36.003 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:14:36.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:14:36.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:14:36.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:14:36.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=687194767366 2024-09-13T15:14:36.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 687194767366 2024-09-13T15:14:36.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-687194767366' 2024-09-13T15:14:36.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:14:36.187 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:14:36.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=712964571138 2024-09-13T15:14:36.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 712964571138 2024-09-13T15:14:36.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-687194767366 1-712964571138' 2024-09-13T15:14:36.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:14:36.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-687194767366 2024-09-13T15:14:36.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:14:36.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:14:36.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-687194767366 2024-09-13T15:14:36.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:14:36.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=687194767366 2024-09-13T15:14:36.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 687194767366' 2024-09-13T15:14:36.333 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 687194767366 2024-09-13T15:14:36.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:14:36.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 687194767364 -lt 687194767366 2024-09-13T15:14:36.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:14:37.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:14:37.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:14:38.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 687194767366 -lt 687194767366 2024-09-13T15:14:38.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:14:38.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-712964571138 2024-09-13T15:14:38.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:14:38.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:14:38.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:14:38.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-712964571138 2024-09-13T15:14:38.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=712964571138 2024-09-13T15:14:38.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 712964571138' 2024-09-13T15:14:38.036 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 712964571138 2024-09-13T15:14:38.036 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:14:38.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 712964571138 -lt 712964571138 2024-09-13T15:14:38.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:14:38.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:14:38.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:14:38.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:14:38.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:14:38.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:14:38.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:14:38.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:14:38.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:14:38.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:14:38.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:14:39.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:14:39.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:14:39.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:14:39.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:14:39.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:14:39.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:14:39.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:14:39.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:14:39.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-09-13T15:14:39.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-09-13T15:14:39.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-13T15:14:39.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:14:39.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-09-13T15:14:39.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-09-13T15:14:39.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-13T15:14:39.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:14:39.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:14:39.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:14:39.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:14:39.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-13T15:14:39.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:14:39.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:14:39.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:14:39.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:14:39.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:14:39.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:14:39.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:14:39.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:14:39.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:14:39.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:14:39.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:14:39.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-13T15:14:39.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:14:39.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:14:39.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:14:39.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:14:39.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:14:39.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-13T15:14:41.176 INFO:tasks.workunit.client.0.smithi112.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-13T15:14:41.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:14:41.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:14:41.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:14:41.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:14:41.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:14:41.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:14:41.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:14:41.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:14:41.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:14:41.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:14:41.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:14:41.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:14:41.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:14:41.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:14:41.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:14:41.797 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:14:41.797 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:14:41.797 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:14:41.797 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:14:41.797 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:14:41.797 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:14:41.797 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:14:41.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:14:41.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:14:41.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:14:41.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:14:41.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:14:41.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:14:41.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:14:41.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:14:41.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:14:41.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:14:41.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:14:41.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:14:41.800 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:14:41.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:14:41.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:14:41.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:14:41.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:14:41.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:14:41.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:14:41.835 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:41.831+0000 7fb13102d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:14:41.837 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:41.835+0000 7fb13102d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:14:41.839 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:41.835+0000 7fb13102d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:14:42.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:14:42.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:14:42.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:14:42.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:14:42.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:14:42.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:42.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:14:42.165 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:14:42.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:42.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:14:42.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:42.858 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:42.855+0000 7fb13102d7c0 -1 Falling back to public interface 2024-09-13T15:14:43.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:43.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:43.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:14:43.504 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:14:43.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:43.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:14:43.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:44.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:44.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:44.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:14:44.833 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:14:44.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:44.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:14:45.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:45.533 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:45.531+0000 7fb13102d7c0 -1 osd.0 168 log_to_monitors true 2024-09-13T15:14:46.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:46.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:46.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:14:46.152 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:14:46.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:46.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:14:46.512 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 172 up_thru 172 down_at 169 last_clean_interval [160,168) [v2:127.0.0.1:6802/2088292847,v1:127.0.0.1:6803/2088292847] [v2:127.0.0.1:6804/2088292847,v1:127.0.0.1:6805/2088292847] exists,up 2e31414c-0818-44ce-b14a-578b74fc94f5 2024-09-13T15:14:46.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:14:46.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:14:46.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:14:46.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:14:46.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:14:46.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:14:46.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:14:46.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:14:46.514 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:14:46.514 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:14:46.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:14:46.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:14:46.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:14:46.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:14:46.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:14:46.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:14:46.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:14:46.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:14:46.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:14:46.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:14:46.955 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:14:46.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:14:46.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:14:46.956 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:14:47.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=738734374914 2024-09-13T15:14:47.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 738734374914 2024-09-13T15:14:47.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-738734374914' 2024-09-13T15:14:47.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:14:47.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:14:47.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=712964571141 2024-09-13T15:14:47.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 712964571141 2024-09-13T15:14:47.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-738734374914 1-712964571141' 2024-09-13T15:14:47.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:14:47.264 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-738734374914 2024-09-13T15:14:47.264 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:14:47.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:14:47.265 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-738734374914 2024-09-13T15:14:47.265 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:14:47.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=738734374914 2024-09-13T15:14:47.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 738734374914' 2024-09-13T15:14:47.266 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 738734374914 2024-09-13T15:14:47.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:14:47.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 687194767366 -lt 738734374914 2024-09-13T15:14:47.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:14:48.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:14:48.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:14:48.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 738734374914 -lt 738734374914 2024-09-13T15:14:48.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:14:48.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-712964571141 2024-09-13T15:14:48.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:14:48.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:14:48.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-712964571141 2024-09-13T15:14:48.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:14:48.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=712964571141 2024-09-13T15:14:48.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 712964571141' 2024-09-13T15:14:48.916 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 712964571141 2024-09-13T15:14:48.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:14:49.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 712964571141 -lt 712964571141 2024-09-13T15:14:49.250 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:14:49.250 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:14:49.250 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:14:49.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:14:49.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:14:49.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:14:49.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:14:49.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:14:49.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:14:49.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:14:49.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:14:50.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:14:50.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:14:50.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:14:50.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:14:50.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:14:50.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:14:50.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:14:50.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-13T15:14:50.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:14:50.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:14:50.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:14:50.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:14:50.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-13T15:14:50.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:14:50.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:14:50.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:14:50.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:14:50.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:14:50.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:14:50.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:14:50.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:14:50.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:14:50.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:14:50.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:14:50.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-13T15:14:50.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:14:50.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:14:50.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:14:50.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:14:50.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:14:50.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-13T15:14:51.834 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-13T15:14:51.834 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-13T15:14:52.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:14:52.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:14:52.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:14:52.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:14:52.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:14:52.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:14:52.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:14:52.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:14:52.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:14:52.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:14:52.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:14:52.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:14:52.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:14:52.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:14:52.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:14:52.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:14:52.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:14:52.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:14:52.441 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:14:52.441 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:14:52.441 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:14:52.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:14:52.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:14:52.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:14:52.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:14:52.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:14:52.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:14:52.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:14:52.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:14:52.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:14:52.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:14:52.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:14:52.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:14:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:14:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:14:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:14:52.443 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:14:52.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:14:52.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:14:52.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:14:52.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:14:52.477 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:52.475+0000 7f058ede47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:14:52.480 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:52.479+0000 7f058ede47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:14:52.482 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:52.479+0000 7f058ede47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:14:52.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:14:52.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:14:52.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:14:52.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:14:52.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:14:52.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:52.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:14:52.772 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:14:52.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:52.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:14:53.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:53.551 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:53.547+0000 7f058ede47c0 -1 Falling back to public interface 2024-09-13T15:14:54.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:54.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:54.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:14:54.119 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:14:54.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:54.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:14:54.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:55.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:55.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:55.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:14:55.457 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:14:55.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:55.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:14:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:14:56.112 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:14:56.111+0000 7f058ede47c0 -1 osd.1 174 log_to_monitors true 2024-09-13T15:14:56.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:14:56.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:14:56.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:14:56.784 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:14:56.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:14:56.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:14:57.388 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 178 up_thru 178 down_at 175 last_clean_interval [166,174) [v2:127.0.0.1:6810/1578484622,v1:127.0.0.1:6811/1578484622] [v2:127.0.0.1:6812/1578484622,v1:127.0.0.1:6813/1578484622] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:14:57.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:14:57.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:14:57.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:14:57.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:14:57.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:14:57.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:14:57.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:14:57.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:14:57.389 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:14:57.389 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:14:57.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:14:57.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:14:57.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:14:57.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:14:57.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:14:57.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:14:57.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:14:57.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:14:57.510 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:14:57.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:14:57.840 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:14:57.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:14:57.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:14:57.840 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:14:58.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=738734374917 2024-09-13T15:14:58.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 738734374917 2024-09-13T15:14:58.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-738734374917' 2024-09-13T15:14:58.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:14:58.007 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:14:58.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=764504178690 2024-09-13T15:14:58.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 764504178690 2024-09-13T15:14:58.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-738734374917 1-764504178690' 2024-09-13T15:14:58.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:14:58.145 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-738734374917 2024-09-13T15:14:58.145 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:14:58.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:14:58.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-738734374917 2024-09-13T15:14:58.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:14:58.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=738734374917 2024-09-13T15:14:58.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 738734374917' 2024-09-13T15:14:58.149 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 738734374917 2024-09-13T15:14:58.150 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:14:58.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 738734374916 -lt 738734374917 2024-09-13T15:14:58.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:14:59.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:14:59.449 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:14:59.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 738734374917 -lt 738734374917 2024-09-13T15:14:59.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:14:59.828 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:14:59.828 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-764504178690 2024-09-13T15:14:59.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:14:59.830 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-764504178690 2024-09-13T15:14:59.831 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:14:59.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=764504178690 2024-09-13T15:14:59.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 764504178690' 2024-09-13T15:14:59.832 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 764504178690 2024-09-13T15:14:59.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:15:00.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 764504178690 -lt 764504178690 2024-09-13T15:15:00.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:15:00.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:15:00.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:15:00.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:15:00.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:15:00.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:15:00.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:15:00.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:15:00.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:15:00.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:15:00.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:15:00.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:15:00.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:15:00.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:15:00.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:15:01.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:15:01.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:15:01.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:15:01.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-09-13T15:15:01.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:15:01.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:15:01.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:15:01.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:15:01.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-09-13T15:15:01.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:15:01.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:15:01.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:15:01.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:15:01.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:15:01.281 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:15:01.281 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:15:01.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:15:01.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:15:01.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:15:01.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:15:01.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-09-13T15:15:01.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:15:01.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:15:01.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:15:01.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:15:01.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:15:01.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-09-13T15:15:02.672 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-13T15:15:02.672 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-13T15:15:03.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:15:03.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:15:03.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:15:03.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:15:03.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:15:03.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:15:03.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:15:03.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:15:03.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:15:03.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:15:03.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:15:03.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:15:03.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:15:03.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:15:03.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:15:03.224 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:15:03.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:15:03.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:15:03.225 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:15:03.225 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:15:03.225 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:15:03.226 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:15:03.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:15:03.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:15:03.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:15:03.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:15:03.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:15:03.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:15:03.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:15:03.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:15:03.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:15:03.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:15:03.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:15:03.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:15:03.228 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:15:03.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:15:03.228 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:15:03.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:15:03.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:15:03.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:15:03.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:15:03.264 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:03.263+0000 7fa309e037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:15:03.267 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:03.263+0000 7fa309e037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:15:03.268 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:03.267+0000 7fa309e037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:15:03.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:15:03.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:15:03.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:15:03.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:15:03.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:15:03.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:03.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:15:03.567 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:15:03.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:03.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:15:03.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:15:04.346 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:04.343+0000 7fa309e037c0 -1 Falling back to public interface 2024-09-13T15:15:04.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:15:04.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:04.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:15:04.984 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:15:04.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:04.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:15:05.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:15:06.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:15:06.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:06.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:15:06.325 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:15:06.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:06.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:15:06.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:15:06.866 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:06.862+0000 7fa309e037c0 -1 osd.1 180 log_to_monitors true 2024-09-13T15:15:07.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:15:07.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:07.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:15:07.662 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:15:07.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:07.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:15:08.112 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 184 up_thru 184 down_at 181 last_clean_interval [178,180) [v2:127.0.0.1:6810/114840457,v1:127.0.0.1:6811/114840457] [v2:127.0.0.1:6812/114840457,v1:127.0.0.1:6813/114840457] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:15:08.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:15:08.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:15:08.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:15:08.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:15:08.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:15:08.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:15:08.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:15:08.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:15:08.115 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:15:08.115 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:15:08.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:15:08.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:15:08.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:15:08.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:15:08.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:15:08.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:15:08.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:15:08.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:15:08.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:15:08.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:15:08.588 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:15:08.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:15:08.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:15:08.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:15:08.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=738734374920 2024-09-13T15:15:08.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 738734374920 2024-09-13T15:15:08.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-738734374920' 2024-09-13T15:15:08.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:15:08.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:15:08.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=790273982466 2024-09-13T15:15:08.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 790273982466 2024-09-13T15:15:08.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-738734374920 1-790273982466' 2024-09-13T15:15:08.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:15:08.883 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-738734374920 2024-09-13T15:15:08.883 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:15:08.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:15:08.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-738734374920 2024-09-13T15:15:08.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:15:08.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=738734374920 2024-09-13T15:15:08.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 738734374920' 2024-09-13T15:15:08.887 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 738734374920 2024-09-13T15:15:08.887 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:15:09.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 738734374919 -lt 738734374920 2024-09-13T15:15:09.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:15:10.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:15:10.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:15:10.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 738734374920 -lt 738734374920 2024-09-13T15:15:10.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:15:10.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:15:10.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-790273982466 2024-09-13T15:15:10.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:15:10.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-790273982466 2024-09-13T15:15:10.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:15:10.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=790273982466 2024-09-13T15:15:10.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 790273982466' 2024-09-13T15:15:10.529 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 790273982466 2024-09-13T15:15:10.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:15:10.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 790273982466 -lt 790273982466 2024-09-13T15:15:10.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:15:10.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:15:10.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:15:11.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:15:11.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:15:11.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:15:11.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:15:11.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:15:11.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:15:11.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:15:11.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:15:11.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:15:11.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:15:11.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:15:11.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:15:11.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:15:11.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:15:11.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:15:11.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-13T15:15:11.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-09-13T15:15:11.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-09-13T15:15:11.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-13T15:15:11.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-13T15:15:11.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-09-13T15:15:11.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2024-09-13T15:15:11.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2024-09-13T15:15:11.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-09-13T15:15:11.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T15:15:12.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-09-13T15:15:12.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-09-13T15:15:12.399 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: {} 2024-09-13T15:15:12.399 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-13T15:15:12.415 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: {} 2024-09-13T15:15:12.416 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-13T15:15:12.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-09-13T15:15:12.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-13T15:15:12.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-13T15:15:12.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-13T15:15:12.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-13T15:15:12.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-13T15:15:12.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-13T15:15:12.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ11 2024-09-13T15:15:12.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-13T15:15:12.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-13T15:15:12.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-13T15:15:12.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-13T15:15:12.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-13T15:15:12.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ11 2024-09-13T15:15:12.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-13T15:15:12.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2024-09-13T15:15:12.431 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-09-13T15:15:12.432 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T15:15:12.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-13T15:15:12.759 INFO:tasks.workunit.client.0.smithi112.stderr:0' 2024-09-13T15:15:12.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-13T15:15:12.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-13T15:15:12.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-13T15:15:12.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-13T15:15:12.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-13T15:15:12.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-13T15:15:12.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-13T15:15:12.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-13T15:15:12.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-13T15:15:12.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-13T15:15:12.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-13T15:15:12.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-13T15:15:12.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-13T15:15:12.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-13T15:15:12.762 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-13T15:15:12.762 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T15:15:12.762 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T15:15:12.762 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:15:12.763 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:15:12.763 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:15:12.763 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.1.asok 2024-09-13T15:15:12.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.63282/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-13T15:15:12.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-13T15:15:12.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-13T15:15:12.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-13T15:15:12.888 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-13T15:15:12.888 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T15:15:12.889 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T15:15:12.889 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:15:12.889 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:15:12.889 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:15:12.889 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.1.asok 2024-09-13T15:15:12.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-13T15:15:12.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.63282/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-09-13T15:15:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-09-13T15:15:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-13T15:15:13.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-13T15:15:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-13T15:15:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-13T15:15:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-13T15:15:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-13T15:15:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ12 2024-09-13T15:15:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-13T15:15:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-13T15:15:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-13T15:15:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-13T15:15:13.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-13T15:15:13.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ12 2024-09-13T15:15:13.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-13T15:15:13.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2024-09-13T15:15:13.003 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-09-13T15:15:13.003 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T15:15:13.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-13T15:15:13.340 INFO:tasks.workunit.client.0.smithi112.stderr:0' 2024-09-13T15:15:13.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-13T15:15:13.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-13T15:15:13.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-13T15:15:13.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-13T15:15:13.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-13T15:15:13.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-13T15:15:13.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-13T15:15:13.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-13T15:15:13.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-13T15:15:13.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-13T15:15:13.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-13T15:15:13.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-13T15:15:13.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-13T15:15:13.343 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-13T15:15:13.343 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-13T15:15:13.343 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T15:15:13.343 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T15:15:13.344 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:15:13.344 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:15:13.345 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:15:13.345 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.0.asok 2024-09-13T15:15:13.345 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.63282/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-13T15:15:13.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-13T15:15:13.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-13T15:15:13.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-13T15:15:13.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-13T15:15:13.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T15:15:13.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T15:15:13.454 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:15:13.454 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:15:13.454 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:15:13.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.0.asok 2024-09-13T15:15:13.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-13T15:15:13.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.63282/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-09-13T15:15:13.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-09-13T15:15:13.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-13T15:15:13.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-13T15:15:13.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-13T15:15:13.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-13T15:15:13.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-13T15:15:13.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-13T15:15:13.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-13T15:15:13.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-13T15:15:13.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-13T15:15:13.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-13T15:15:13.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-13T15:15:13.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-13T15:15:13.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-13T15:15:13.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-13T15:15:13.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-13T15:15:13.572 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T15:15:13.572 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-13T15:15:13.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-13T15:15:13.910 INFO:tasks.workunit.client.0.smithi112.stderr:0' 2024-09-13T15:15:13.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-13T15:15:13.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-13T15:15:13.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-13T15:15:13.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-13T15:15:13.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-13T15:15:13.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-13T15:15:13.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-13T15:15:13.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-13T15:15:13.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-13T15:15:13.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-13T15:15:13.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-13T15:15:13.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-13T15:15:13.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-13T15:15:13.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-13T15:15:13.912 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-13T15:15:13.913 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T15:15:13.913 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T15:15:13.913 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:15:13.913 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:15:13.913 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:15:13.914 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.0.asok 2024-09-13T15:15:13.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.63282/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-13T15:15:14.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-13T15:15:14.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-13T15:15:14.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-13T15:15:14.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-13T15:15:14.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T15:15:14.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T15:15:14.033 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:15:14.033 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:15:14.033 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:15:14.033 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.0.asok 2024-09-13T15:15:14.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-13T15:15:14.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.63282/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-09-13T15:15:14.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-09-13T15:15:14.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-13T15:15:14.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-13T15:15:14.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-13T15:15:14.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-13T15:15:14.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-13T15:15:14.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-13T15:15:14.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-13T15:15:14.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-13T15:15:14.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-13T15:15:14.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-13T15:15:14.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-13T15:15:14.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-13T15:15:14.145 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-13T15:15:14.145 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-13T15:15:14.145 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-13T15:15:14.145 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T15:15:14.145 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-13T15:15:14.467 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-13T15:15:14.467 INFO:tasks.workunit.client.0.smithi112.stderr:0' 2024-09-13T15:15:14.467 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-13T15:15:14.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-13T15:15:14.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-13T15:15:14.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-13T15:15:14.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-13T15:15:14.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-13T15:15:14.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-13T15:15:14.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-13T15:15:14.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-13T15:15:14.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-13T15:15:14.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-13T15:15:14.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-13T15:15:14.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-13T15:15:14.470 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-13T15:15:14.471 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-13T15:15:14.471 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T15:15:14.471 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T15:15:14.471 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:15:14.471 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:15:14.471 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:15:14.472 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.1.asok 2024-09-13T15:15:14.472 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.63282/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-13T15:15:14.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-13T15:15:14.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-13T15:15:14.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-13T15:15:14.581 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-13T15:15:14.581 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T15:15:14.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T15:15:14.582 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:15:14.583 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:15:14.583 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:15:14.583 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.1.asok 2024-09-13T15:15:14.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-13T15:15:14.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.63282/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-09-13T15:15:14.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-09-13T15:15:14.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=3.0 2024-09-13T15:15:14.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 3.0 2024-09-13T15:15:14.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:15:14.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:15:14.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:15:14.703 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:15:15.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-13T15:10:48.448335+0000 2024-09-13T15:15:15.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 3.0 2024-09-13T15:15:15.338 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 3.0 on osd.1 to scrub 2024-09-13T15:15:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 3.0 2024-09-13T15:10:48.448335+0000 2024-09-13T15:15:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-13T15:15:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:10:48.448335+0000 2024-09-13T15:15:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:15:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:15:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:15:15.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-13T15:15:15.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:15:15.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:15:15.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:15:15.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:15:15.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:10:48.448335+0000 '>' 2024-09-13T15:10:48.448335+0000 2024-09-13T15:15:15.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:15:16.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:15:16.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:15:16.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-13T15:15:16.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:15:16.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:15:16.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:15:16.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:15:17.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:10:48.448335+0000 '>' 2024-09-13T15:10:48.448335+0000 2024-09-13T15:15:17.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:15:18.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:15:18.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:15:18.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-13T15:15:18.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:15:18.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:15:18.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:15:18.007 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:15:18.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:15:15.817427+0000 '>' 2024-09-13T15:10:48.448335+0000 2024-09-13T15:15:18.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:15:18.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-09-13T15:15:18.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-09-13T15:15:18.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-09-13T15:15:18.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-09-13T15:15:18.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-09-13T15:15:18.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-09-13T15:15:18.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-09-13T15:15:18.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-09-13T15:15:18.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-09-13T15:15:18.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-09-13T15:15:18.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-09-13T15:15:18.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-09-13T15:15:18.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-09-13T15:15:18.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-09-13T15:15:18.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-09-13T15:15:18.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-09-13T15:15:18.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-09-13T15:15:18.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-09-13T15:15:18.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-09-13T15:15:18.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2024-09-13T15:15:18.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-09-13T15:15:18.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-09-13T15:15:18.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-09-13T15:15:18.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-09-13T15:15:18.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:15:18.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-09-13T15:15:18.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-09-13T15:15:18.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-09-13T15:15:18.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-09-13T15:15:18.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-13T15:15:18.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-09-13T15:15:18.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-09-13T15:15:18.712 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-09-13T15:15:18.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=184 2024-09-13T15:15:18.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-13T15:15:18.741 INFO:tasks.workunit.client.0.smithi112.stderr: . as $in 2024-09-13T15:15:18.741 INFO:tasks.workunit.client.0.smithi112.stderr: | if type == "object" then 2024-09-13T15:15:18.741 INFO:tasks.workunit.client.0.smithi112.stderr: reduce keys[] as $key 2024-09-13T15:15:18.741 INFO:tasks.workunit.client.0.smithi112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-13T15:15:18.741 INFO:tasks.workunit.client.0.smithi112.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-13T15:15:18.741 INFO:tasks.workunit.client.0.smithi112.stderr: else f 2024-09-13T15:15:18.741 INFO:tasks.workunit.client.0.smithi112.stderr: end; 2024-09-13T15:15:18.742 INFO:tasks.workunit.client.0.smithi112.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-13T15:15:18.742 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-13T15:15:18.742 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-13T15:15:18.742 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-13T15:15:18.742 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-13T15:15:18.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-13T15:15:18.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-13T15:15:18.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-13T15:15:18.789 INFO:tasks.workunit.client.0.smithi112.stderr: . as $in 2024-09-13T15:15:18.789 INFO:tasks.workunit.client.0.smithi112.stderr: | if type == "object" then 2024-09-13T15:15:18.789 INFO:tasks.workunit.client.0.smithi112.stderr: reduce keys[] as $key 2024-09-13T15:15:18.789 INFO:tasks.workunit.client.0.smithi112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-13T15:15:18.790 INFO:tasks.workunit.client.0.smithi112.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-13T15:15:18.790 INFO:tasks.workunit.client.0.smithi112.stderr: else f 2024-09-13T15:15:18.790 INFO:tasks.workunit.client.0.smithi112.stderr: end; 2024-09-13T15:15:18.790 INFO:tasks.workunit.client.0.smithi112.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-13T15:15:18.790 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-13T15:15:18.790 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-13T15:15:18.790 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-13T15:15:18.790 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-13T15:15:18.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-13T15:15:18.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-13T15:15:18.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-13T15:15:18.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-13T15:15:18.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-09-13T15:15:18.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-09-13T15:15:18.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-09-13T15:15:18.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-09-13T15:15:18.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-09-13T15:15:19.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-13T15:15:19.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:15:19.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:15:19.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:15:19.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:15:19.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-13T15:15:19.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:15:19.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:15:19.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:15:19.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:15:19.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:15:19.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:15:19.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:15:19.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:15:19.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:15:19.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:15:20.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:15:20.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-13T15:15:20.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:15:20.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:15:20.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:15:20.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:15:20.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:15:20.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-13T15:15:21.626 INFO:tasks.workunit.client.0.smithi112.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-13T15:15:22.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:15:22.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:15:22.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:15:22.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:15:22.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:15:22.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:15:22.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:15:22.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:15:22.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:15:22.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:15:22.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:15:22.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:15:22.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:15:22.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:15:22.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:15:22.164 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:15:22.164 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:15:22.164 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:15:22.164 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:15:22.164 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:15:22.165 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:15:22.165 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:15:22.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:15:22.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:15:22.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:15:22.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:15:22.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:15:22.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:15:22.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:15:22.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:15:22.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:15:22.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:15:22.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:15:22.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:15:22.168 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:15:22.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:15:22.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:15:22.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:15:22.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:15:22.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:15:22.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:15:22.201 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:22.198+0000 7f005001f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:15:22.204 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:22.202+0000 7f005001f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:15:22.206 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:22.202+0000 7f005001f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:15:22.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:15:22.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:15:22.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:15:22.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:15:22.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:15:22.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:22.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:15:22.502 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:15:22.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:22.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:15:22.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:15:23.462 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:23.458+0000 7f005001f7c0 -1 Falling back to public interface 2024-09-13T15:15:23.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:15:23.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:23.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:15:23.822 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:15:23.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:23.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:15:24.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:15:25.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:15:25.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:25.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:15:25.173 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:15:25.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:25.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:15:25.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:15:26.403 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:26.398+0000 7f005001f7c0 -1 osd.0 186 log_to_monitors true 2024-09-13T15:15:26.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:15:26.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:26.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:15:26.512 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:15:26.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:26.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:15:26.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:15:27.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:15:27.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:27.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:15:27.864 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:15:27.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:27.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:15:28.402 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 190 up_thru 190 down_at 187 last_clean_interval [172,186) [v2:127.0.0.1:6802/659665926,v1:127.0.0.1:6803/659665926] [v2:127.0.0.1:6804/659665926,v1:127.0.0.1:6805/659665926] exists,up 2e31414c-0818-44ce-b14a-578b74fc94f5 2024-09-13T15:15:28.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:15:28.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:15:28.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:15:28.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:15:28.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:15:28.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:15:28.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:15:28.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:15:28.402 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:15:28.402 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:15:28.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:15:28.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:15:28.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:15:28.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:15:28.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:15:28.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:15:28.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:15:28.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:15:28.510 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:15:28.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:15:28.824 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:15:28.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:15:28.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:15:28.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:15:28.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=816043786242 2024-09-13T15:15:28.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 816043786242 2024-09-13T15:15:28.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-816043786242' 2024-09-13T15:15:28.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:15:28.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:15:29.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=790273982471 2024-09-13T15:15:29.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 790273982471 2024-09-13T15:15:29.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-816043786242 1-790273982471' 2024-09-13T15:15:29.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:15:29.107 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-816043786242 2024-09-13T15:15:29.107 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:15:29.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:15:29.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-816043786242 2024-09-13T15:15:29.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:15:29.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=816043786242 2024-09-13T15:15:29.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 816043786242' 2024-09-13T15:15:29.110 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 816043786242 2024-09-13T15:15:29.110 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:15:29.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 816043786242 2024-09-13T15:15:29.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:15:30.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:15:30.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:15:30.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 816043786242 -lt 816043786242 2024-09-13T15:15:30.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:15:30.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-790273982471 2024-09-13T15:15:30.757 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:15:30.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:15:30.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-790273982471 2024-09-13T15:15:30.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:15:30.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=790273982471 2024-09-13T15:15:30.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 790273982471' 2024-09-13T15:15:30.762 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 790273982471 2024-09-13T15:15:30.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:15:31.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 790273982471 -lt 790273982471 2024-09-13T15:15:31.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:15:31.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:15:31.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:15:31.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:15:31.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:15:31.455 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:15:31.455 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:15:31.455 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:15:31.455 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:15:31.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:15:31.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:15:31.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:15:31.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:15:31.779 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:15:31.779 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:15:32.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:15:32.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:15:32.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:15:32.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-09-13T15:15:32.193 INFO:tasks.workunit.client.0.smithi112.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-09-13T15:15:32.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-09-13T15:15:32.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-09-13T15:15:32.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:15:32.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:15:32.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:15:32.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:15:32.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-09-13T15:15:32.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:15:32.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:15:32.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:15:32.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:15:32.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:15:32.194 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:15:32.194 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:15:32.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:15:32.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:15:32.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:15:33.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:15:33.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-09-13T15:15:33.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:15:33.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:15:33.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:15:33.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:15:33.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:15:33.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-09-13T15:15:34.692 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-13T15:15:34.692 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-13T15:15:35.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:15:35.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:15:35.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:15:35.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:15:35.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:15:35.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:15:35.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:15:35.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:15:35.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:15:35.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:15:35.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:15:35.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:15:35.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:15:35.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:15:35.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:15:35.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:15:35.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:15:35.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:15:35.232 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:15:35.232 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:15:35.232 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:15:35.232 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:15:35.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:15:35.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:15:35.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:15:35.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:15:35.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:15:35.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:15:35.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:15:35.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:15:35.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:15:35.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:15:35.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:15:35.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:15:35.234 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:15:35.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:15:35.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:15:35.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:15:35.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:15:35.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:15:35.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:15:35.268 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:35.262+0000 7f33e75c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:15:35.270 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:35.266+0000 7f33e75c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:15:35.271 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:35.270+0000 7f33e75c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:15:35.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:15:35.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:15:35.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:15:35.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:15:35.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:15:35.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:35.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:15:35.562 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:15:35.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:35.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:15:35.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:15:36.570 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:36.566+0000 7f33e75c47c0 -1 Falling back to public interface 2024-09-13T15:15:36.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:15:36.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:36.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:15:36.929 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:15:36.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:36.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:15:37.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:15:38.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:15:38.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:38.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:15:38.257 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:15:38.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:38.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:15:38.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:15:39.047 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:39.042+0000 7f33e75c47c0 -1 osd.1 192 log_to_monitors true 2024-09-13T15:15:39.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:15:39.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:39.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:15:39.599 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:15:39.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:39.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:15:39.931 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 196 up_thru 190 down_at 193 last_clean_interval [184,192) [v2:127.0.0.1:6810/2102059249,v1:127.0.0.1:6811/2102059249] [v2:127.0.0.1:6812/2102059249,v1:127.0.0.1:6813/2102059249] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:15:39.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:15:39.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:15:39.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:15:39.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:15:39.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:15:39.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:15:39.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:15:39.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:15:39.933 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:15:39.933 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:15:39.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:15:39.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:15:39.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:15:40.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:15:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:15:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:15:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:15:40.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:15:40.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:15:40.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:15:40.380 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:15:40.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:15:40.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:15:40.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:15:40.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=816043786245 2024-09-13T15:15:40.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 816043786245 2024-09-13T15:15:40.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-816043786245' 2024-09-13T15:15:40.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:15:40.652 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:15:40.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=841813590018 2024-09-13T15:15:40.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 841813590018 2024-09-13T15:15:40.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-816043786245 1-841813590018' 2024-09-13T15:15:40.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:15:40.798 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-816043786245 2024-09-13T15:15:40.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:15:40.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:15:40.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-816043786245 2024-09-13T15:15:40.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:15:40.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=816043786245 2024-09-13T15:15:40.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 816043786245' 2024-09-13T15:15:40.803 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 816043786245 2024-09-13T15:15:40.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:15:41.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 816043786244 -lt 816043786245 2024-09-13T15:15:41.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:15:42.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:15:42.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:15:42.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 816043786245 -lt 816043786245 2024-09-13T15:15:42.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:15:42.544 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-841813590018 2024-09-13T15:15:42.544 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:15:42.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:15:42.546 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-841813590018 2024-09-13T15:15:42.546 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:15:42.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=841813590018 2024-09-13T15:15:42.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 841813590018' 2024-09-13T15:15:42.548 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 841813590018 2024-09-13T15:15:42.549 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:15:42.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 841813590018 -lt 841813590018 2024-09-13T15:15:42.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:15:42.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:15:42.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:15:43.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:15:43.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:15:43.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:15:43.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:15:43.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:15:43.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:15:43.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:15:43.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:15:43.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:15:43.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:15:43.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:15:43.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:15:44.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:15:44.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:15:44.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:15:44.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-09-13T15:15:44.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-13T15:15:44.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:15:44.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:15:44.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:15:44.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:15:44.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-13T15:15:44.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:15:44.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:15:44.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:15:44.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:15:44.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:15:44.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:15:44.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:15:44.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:15:44.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:15:44.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:15:45.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:15:45.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-13T15:15:45.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:15:45.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:15:45.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:15:45.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:15:45.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:15:45.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-13T15:15:46.940 INFO:tasks.workunit.client.0.smithi112.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-13T15:15:47.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:15:47.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:15:47.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:15:47.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:15:47.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:15:47.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:15:47.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:15:47.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:15:47.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:15:47.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:15:47.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:15:47.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:15:47.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:15:47.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:15:47.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:15:47.477 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:15:47.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:15:47.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:15:47.478 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:15:47.478 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:15:47.478 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:15:47.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:15:47.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:15:47.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:15:47.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:15:47.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:15:47.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:15:47.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:15:47.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:15:47.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:15:47.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:15:47.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:15:47.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:15:47.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:15:47.479 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:15:47.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:15:47.480 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:15:47.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:15:47.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:15:47.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:15:47.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:15:47.516 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:47.509+0000 7f261b29c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:15:47.518 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:47.513+0000 7f261b29c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:15:47.520 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:47.517+0000 7f261b29c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:15:47.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:15:47.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:15:47.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:15:47.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:15:47.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:15:47.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:47.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:15:47.821 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:15:47.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:47.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:15:48.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:15:48.567 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:48.561+0000 7f261b29c7c0 -1 Falling back to public interface 2024-09-13T15:15:49.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:15:49.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:49.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:15:49.143 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:15:49.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:49.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:15:49.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:15:50.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:15:50.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:50.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:15:50.477 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:15:50.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:50.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:15:50.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:15:51.380 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:15:51.377+0000 7f261b29c7c0 -1 osd.0 198 log_to_monitors true 2024-09-13T15:15:51.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:15:51.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:15:51.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:15:51.808 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:15:51.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:15:51.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:15:52.212 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 202 up_thru 197 down_at 199 last_clean_interval [190,198) [v2:127.0.0.1:6802/963157420,v1:127.0.0.1:6803/963157420] [v2:127.0.0.1:6804/963157420,v1:127.0.0.1:6805/963157420] exists,up 2e31414c-0818-44ce-b14a-578b74fc94f5 2024-09-13T15:15:52.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:15:52.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:15:52.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:15:52.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:15:52.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:15:52.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:15:52.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:15:52.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:15:52.213 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:15:52.213 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:15:52.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:15:52.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:15:52.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:15:52.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:15:52.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:15:52.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:15:52.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:15:52.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:15:52.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:15:52.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:15:52.827 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:15:52.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:15:52.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:15:52.828 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:15:53.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=867583393794 2024-09-13T15:15:53.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 867583393794 2024-09-13T15:15:53.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-867583393794' 2024-09-13T15:15:53.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:15:53.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:15:53.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=841813590021 2024-09-13T15:15:53.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 841813590021 2024-09-13T15:15:53.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-867583393794 1-841813590021' 2024-09-13T15:15:53.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:15:53.249 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-867583393794 2024-09-13T15:15:53.249 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:15:53.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:15:53.251 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-867583393794 2024-09-13T15:15:53.251 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:15:53.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=867583393794 2024-09-13T15:15:53.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 867583393794' 2024-09-13T15:15:53.253 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 867583393794 2024-09-13T15:15:53.253 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:15:53.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 816043786246 -lt 867583393794 2024-09-13T15:15:53.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:15:54.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:15:54.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:15:54.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 867583393794 -lt 867583393794 2024-09-13T15:15:54.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:15:54.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-841813590021 2024-09-13T15:15:54.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:15:54.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:15:54.906 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-841813590021 2024-09-13T15:15:54.906 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:15:54.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=841813590021 2024-09-13T15:15:54.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 841813590021' 2024-09-13T15:15:54.908 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 841813590021 2024-09-13T15:15:54.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:15:55.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 841813590021 -lt 841813590021 2024-09-13T15:15:55.236 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:15:55.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:15:55.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:15:55.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:15:55.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:15:55.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:15:55.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:15:55.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:15:55.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:15:55.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:15:55.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:15:56.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:15:56.030 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:15:56.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:15:56.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:15:56.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-13T15:15:56.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-13T15:15:56.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:15:56.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-13T15:15:56.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:15:56.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:15:56.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:15:56.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:15:56.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:15:56.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:15:56.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:15:56.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:15:56.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:15:56.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:15:56.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:15:56.845 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:15:56.845 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:15:56.845 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:15:57.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-13T15:15:57.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-13T15:15:57.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:15:57.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:15:57.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:15:57.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:15:57.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:15:57.223 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:15:57.224 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:15:57.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:15:57.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:15:57.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T15:15:57.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:15:57.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T15:15:57.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:15:57.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:15:57.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:15:57.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:15:57.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:15:57.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:15:57.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:15:57.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:15:58.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:15:58.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:15:58.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:15:58.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:15:58.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:15:58.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:15:58.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:15:58.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-13T15:15:58.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:15:58.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:15:58.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:15:58.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:15:58.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-13T15:15:58.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:15:58.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:15:58.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:15:58.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:15:58.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:15:58.554 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:15:58.554 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:15:58.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:15:58.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:15:58.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:15:58.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:15:58.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-13T15:15:58.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:15:58.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:15:58.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:15:58.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:15:58.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:15:58.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-13T15:16:00.183 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-13T15:16:00.183 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-13T15:16:00.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:16:00.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:16:00.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:16:00.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:16:00.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:16:00.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:16:00.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:16:00.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:16:00.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:16:00.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:16:00.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:16:00.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:16:00.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:16:00.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:16:00.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:16:00.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:16:00.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:16:00.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:16:00.722 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:16:00.722 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:00.723 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:00.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:16:00.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:16:00.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:16:00.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:16:00.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:16:00.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:16:00.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:16:00.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:16:00.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:16:00.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:16:00.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:16:00.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:16:00.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:16:00.724 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:16:00.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:16:00.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:16:00.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:16:00.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:16:00.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:16:00.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:16:00.759 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:16:00.753+0000 7f47558a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:16:00.761 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:16:00.757+0000 7f47558a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:16:00.763 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:16:00.757+0000 7f47558a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:16:01.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:16:01.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:16:01.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:16:01.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:16:01.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:16:01.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:01.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:16:01.061 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:16:01.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:01.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:16:01.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:02.243 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:16:02.237+0000 7f47558a27c0 -1 Falling back to public interface 2024-09-13T15:16:02.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:02.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:02.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:16:02.393 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:16:02.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:02.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:16:02.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:03.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:03.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:03.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:16:03.708 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:16:03.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:03.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:16:04.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:04.741 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:16:04.737+0000 7f47558a27c0 -1 osd.1 204 log_to_monitors true 2024-09-13T15:16:05.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:05.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:05.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:16:05.033 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:16:05.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:05.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:16:05.382 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 208 up_thru 202 down_at 205 last_clean_interval [196,204) [v2:127.0.0.1:6810/2903469387,v1:127.0.0.1:6811/2903469387] [v2:127.0.0.1:6812/2903469387,v1:127.0.0.1:6813/2903469387] exists,up f2c8290b-0073-4005-a233-4452b6605eb6 2024-09-13T15:16:05.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:16:05.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:16:05.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:16:05.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:16:05.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:16:05.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:16:05.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:16:05.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:16:05.384 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:16:05.384 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:16:05.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:16:05.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:16:05.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:16:05.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:16:05.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:16:05.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:16:05.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:16:05.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:16:05.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:16:05.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:16:05.918 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:16:05.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:16:05.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:16:05.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:16:06.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=867583393797 2024-09-13T15:16:06.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 867583393797 2024-09-13T15:16:06.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-867583393797' 2024-09-13T15:16:06.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:16:06.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:16:06.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=893353197570 2024-09-13T15:16:06.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 893353197570 2024-09-13T15:16:06.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-867583393797 1-893353197570' 2024-09-13T15:16:06.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:16:06.279 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-867583393797 2024-09-13T15:16:06.279 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:16:06.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:16:06.280 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-867583393797 2024-09-13T15:16:06.280 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:16:06.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=867583393797 2024-09-13T15:16:06.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 867583393797' 2024-09-13T15:16:06.282 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 867583393797 2024-09-13T15:16:06.282 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:16:06.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 867583393796 -lt 867583393797 2024-09-13T15:16:06.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:16:07.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:16:07.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:16:07.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 867583393798 -lt 867583393797 2024-09-13T15:16:07.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:16:07.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-893353197570 2024-09-13T15:16:07.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:16:07.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:16:07.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:16:07.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-893353197570 2024-09-13T15:16:07.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=893353197570 2024-09-13T15:16:07.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 893353197570' 2024-09-13T15:16:07.930 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 893353197570 2024-09-13T15:16:07.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:16:08.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 893353197570 -lt 893353197570 2024-09-13T15:16:08.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:16:08.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:16:08.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:16:08.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:16:08.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:16:08.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:16:08.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:16:08.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:16:08.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:16:08.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:16:08.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:16:08.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:16:08.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:16:08.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:16:08.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:16:09.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:16:09.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:16:09.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:16:09.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-09-13T15:16:09.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-09-13T15:16:09.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-13T15:16:09.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-13T15:16:09.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-13T15:16:09.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-13T15:16:09.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-13T15:16:09.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-13T15:16:09.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ11 2024-09-13T15:16:09.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-13T15:16:09.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-13T15:16:09.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-13T15:16:09.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-13T15:16:09.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-13T15:16:09.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ11 2024-09-13T15:16:09.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-13T15:16:09.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2024-09-13T15:16:09.347 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-09-13T15:16:09.347 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T15:16:09.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-13T15:16:09.701 INFO:tasks.workunit.client.0.smithi112.stderr:0' 2024-09-13T15:16:09.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-13T15:16:09.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-13T15:16:09.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-13T15:16:09.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-13T15:16:09.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-13T15:16:09.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-13T15:16:09.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-13T15:16:09.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-13T15:16:09.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-13T15:16:09.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-13T15:16:09.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-13T15:16:09.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-13T15:16:09.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-13T15:16:09.704 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-13T15:16:09.705 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-13T15:16:09.705 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T15:16:09.706 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T15:16:09.706 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:16:09.706 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:09.706 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:09.706 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.1.asok 2024-09-13T15:16:09.706 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.63282/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-13T15:16:09.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-13T15:16:09.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-13T15:16:09.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-13T15:16:09.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-13T15:16:09.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T15:16:09.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T15:16:09.818 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:16:09.818 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:09.818 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:09.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.1.asok 2024-09-13T15:16:09.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-13T15:16:09.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.63282/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-09-13T15:16:09.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-09-13T15:16:09.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-13T15:16:09.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-13T15:16:09.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-13T15:16:09.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-13T15:16:09.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-13T15:16:09.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-13T15:16:09.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ12 2024-09-13T15:16:09.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-13T15:16:09.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-13T15:16:09.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-13T15:16:09.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-13T15:16:09.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-13T15:16:09.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ12 2024-09-13T15:16:09.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-13T15:16:09.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2024-09-13T15:16:09.929 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-09-13T15:16:09.929 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T15:16:10.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-13T15:16:10.235 INFO:tasks.workunit.client.0.smithi112.stderr:0' 2024-09-13T15:16:10.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-13T15:16:10.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-13T15:16:10.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-13T15:16:10.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-13T15:16:10.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-13T15:16:10.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-13T15:16:10.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-13T15:16:10.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-13T15:16:10.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-13T15:16:10.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-13T15:16:10.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-13T15:16:10.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-13T15:16:10.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-13T15:16:10.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-13T15:16:10.238 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-13T15:16:10.238 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T15:16:10.238 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T15:16:10.239 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:16:10.239 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:10.239 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:10.239 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.0.asok 2024-09-13T15:16:10.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.63282/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-13T15:16:10.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-13T15:16:10.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-13T15:16:10.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-13T15:16:10.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-13T15:16:10.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T15:16:10.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T15:16:10.356 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:16:10.356 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:10.356 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:10.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.0.asok 2024-09-13T15:16:10.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-13T15:16:10.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.63282/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-09-13T15:16:10.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-09-13T15:16:10.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-13T15:16:10.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-13T15:16:10.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-13T15:16:10.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-13T15:16:10.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-13T15:16:10.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-13T15:16:10.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-13T15:16:10.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-13T15:16:10.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-13T15:16:10.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-13T15:16:10.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-13T15:16:10.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-13T15:16:10.469 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-13T15:16:10.470 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-13T15:16:10.470 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-13T15:16:10.470 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-13T15:16:10.470 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T15:16:10.789 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-13T15:16:10.790 INFO:tasks.workunit.client.0.smithi112.stderr:0' 2024-09-13T15:16:10.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-13T15:16:10.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-13T15:16:10.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-13T15:16:10.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-13T15:16:10.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-13T15:16:10.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-13T15:16:10.791 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-13T15:16:10.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-13T15:16:10.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-13T15:16:10.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-13T15:16:10.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-13T15:16:10.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-13T15:16:10.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-13T15:16:10.794 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-13T15:16:10.794 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T15:16:10.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-13T15:16:10.794 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T15:16:10.794 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:16:10.794 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:10.794 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:10.795 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.0.asok 2024-09-13T15:16:10.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.63282/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-13T15:16:10.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-13T15:16:10.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-13T15:16:10.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-13T15:16:10.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-13T15:16:10.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T15:16:10.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T15:16:10.905 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:16:10.905 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:10.905 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:10.906 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.0.asok 2024-09-13T15:16:10.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-13T15:16:10.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.63282/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-09-13T15:16:11.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-09-13T15:16:11.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-13T15:16:11.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-13T15:16:11.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-13T15:16:11.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-13T15:16:11.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-13T15:16:11.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-13T15:16:11.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-13T15:16:11.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-13T15:16:11.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-13T15:16:11.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-13T15:16:11.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-13T15:16:11.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-13T15:16:11.016 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-13T15:16:11.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-13T15:16:11.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-13T15:16:11.017 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-13T15:16:11.017 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T15:16:11.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-13T15:16:11.316 INFO:tasks.workunit.client.0.smithi112.stderr:0' 2024-09-13T15:16:11.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-13T15:16:11.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-13T15:16:11.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-13T15:16:11.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-13T15:16:11.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-13T15:16:11.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-13T15:16:11.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-13T15:16:11.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-13T15:16:11.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-13T15:16:11.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-13T15:16:11.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-13T15:16:11.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-13T15:16:11.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-13T15:16:11.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-13T15:16:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-13T15:16:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T15:16:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T15:16:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:16:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:11.320 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.1.asok 2024-09-13T15:16:11.320 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.63282/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-13T15:16:11.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-13T15:16:11.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-13T15:16:11.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-13T15:16:11.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T15:16:11.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T15:16:11.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-13T15:16:11.429 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:16:11.429 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:11.429 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:11.429 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-osd.1.asok 2024-09-13T15:16:11.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-13T15:16:11.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.63282/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-09-13T15:16:11.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-09-13T15:16:11.670 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: {} 2024-09-13T15:16:11.670 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0: osd_max_object_size = '' 2024-09-13T15:16:11.686 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: {} 2024-09-13T15:16:11.686 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1: osd_max_object_size = '' 2024-09-13T15:16:11.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-09-13T15:16:11.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-13T15:16:11.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:16:11.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:16:11.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:16:11.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:16:11.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:16:12.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-13T15:10:48.448335+0000 2024-09-13T15:16:12.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-13T15:16:12.311 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-09-13T15:16:12.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-13T15:10:48.448335+0000 last_deep_scrub_stamp 2024-09-13T15:16:12.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-13T15:16:12.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:10:48.448335+0000 2024-09-13T15:16:12.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-13T15:16:12.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:16:12.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:16:12.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:16:12.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:16:12.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:16:12.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:16:12.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:16:12.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:10:48.448335+0000 '>' 2024-09-13T15:10:48.448335+0000 2024-09-13T15:16:12.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:16:13.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:16:13.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:16:13.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:16:13.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:16:13.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:16:13.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:16:13.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:16:13.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:10:48.448335+0000 '>' 2024-09-13T15:10:48.448335+0000 2024-09-13T15:16:13.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:16:14.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:16:14.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:16:14.944 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:16:14.944 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:16:14.944 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:16:14.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:16:14.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:16:15.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:10:48.448335+0000 '>' 2024-09-13T15:10:48.448335+0000 2024-09-13T15:16:15.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:16:16.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:16:16.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:16:16.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-13T15:16:16.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:16:16.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T15:16:16.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:16:16.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-13T15:16:16.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:16:13.050000+0000 '>' 2024-09-13T15:10:48.448335+0000 2024-09-13T15:16:16.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:16:16.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-09-13T15:16:16.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-09-13T15:16:16.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-09-13T15:16:16.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-09-13T15:16:16.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-09-13T15:16:16.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-09-13T15:16:16.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-09-13T15:16:16.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-09-13T15:16:16.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-09-13T15:16:16.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-09-13T15:16:16.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-09-13T15:16:16.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-09-13T15:16:16.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-09-13T15:16:16.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-09-13T15:16:16.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-09-13T15:16:16.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-09-13T15:16:16.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-09-13T15:16:16.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-09-13T15:16:16.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-09-13T15:16:16.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-09-13T15:16:16.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-09-13T15:16:16.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-09-13T15:16:16.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-09-13T15:16:16.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-09-13T15:16:16.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-09-13T15:16:16.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-09-13T15:16:16.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-09-13T15:16:16.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-09-13T15:16:16.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-09-13T15:16:16.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-09-13T15:16:16.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-09-13T15:16:16.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-09-13T15:16:16.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-09-13T15:16:16.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-09-13T15:16:16.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-09-13T15:16:16.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-09-13T15:16:16.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2024-09-13T15:16:16.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-09-13T15:16:16.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-09-13T15:16:16.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-09-13T15:16:16.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:16.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:16.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:17.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:17.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:17.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:17.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:17.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:17.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:17.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-13T15:16:17.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-09-13T15:16:17.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-09-13T15:16:17.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-09-13T15:16:17.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-09-13T15:16:17.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-13T15:16:17.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-09-13T15:16:17.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-09-13T15:16:17.793 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-09-13T15:16:17.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=208 2024-09-13T15:16:17.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-13T15:16:17.821 INFO:tasks.workunit.client.0.smithi112.stderr: . as $in 2024-09-13T15:16:17.821 INFO:tasks.workunit.client.0.smithi112.stderr: | if type == "object" then 2024-09-13T15:16:17.821 INFO:tasks.workunit.client.0.smithi112.stderr: reduce keys[] as $key 2024-09-13T15:16:17.821 INFO:tasks.workunit.client.0.smithi112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-13T15:16:17.821 INFO:tasks.workunit.client.0.smithi112.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-13T15:16:17.821 INFO:tasks.workunit.client.0.smithi112.stderr: else f 2024-09-13T15:16:17.821 INFO:tasks.workunit.client.0.smithi112.stderr: end; 2024-09-13T15:16:17.821 INFO:tasks.workunit.client.0.smithi112.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-13T15:16:17.821 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-13T15:16:17.822 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-13T15:16:17.822 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-13T15:16:17.822 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-13T15:16:17.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-13T15:16:17.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-13T15:16:17.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-13T15:16:17.873 INFO:tasks.workunit.client.0.smithi112.stderr: . as $in 2024-09-13T15:16:17.873 INFO:tasks.workunit.client.0.smithi112.stderr: | if type == "object" then 2024-09-13T15:16:17.873 INFO:tasks.workunit.client.0.smithi112.stderr: reduce keys[] as $key 2024-09-13T15:16:17.873 INFO:tasks.workunit.client.0.smithi112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-13T15:16:17.873 INFO:tasks.workunit.client.0.smithi112.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-13T15:16:17.873 INFO:tasks.workunit.client.0.smithi112.stderr: else f 2024-09-13T15:16:17.873 INFO:tasks.workunit.client.0.smithi112.stderr: end; 2024-09-13T15:16:17.874 INFO:tasks.workunit.client.0.smithi112.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-13T15:16:17.874 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-13T15:16:17.874 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-13T15:16:17.874 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-13T15:16:17.874 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-13T15:16:17.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-13T15:16:17.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-13T15:16:17.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-13T15:16:17.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-13T15:16:17.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-09-13T15:16:17.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-09-13T15:16:17.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-09-13T15:16:17.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=3.0 2024-09-13T15:16:17.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 3.0 2024-09-13T15:16:17.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:16:17.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:16:17.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:16:17.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:16:18.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T15:16:13.050000+0000 2024-09-13T15:16:18.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 3.0 2024-09-13T15:16:18.522 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 3.0 on osd.1 to repair 2024-09-13T15:16:18.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 3.0 2024-09-13T15:16:13.050000+0000 2024-09-13T15:16:18.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-13T15:16:18.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:16:13.050000+0000 2024-09-13T15:16:18.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:16:18.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:16:18.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:16:18.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-13T15:16:18.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:16:18.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:16:18.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:16:18.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:16:18.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:16:13.050000+0000 '>' 2024-09-13T15:16:13.050000+0000 2024-09-13T15:16:18.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:16:19.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:16:19.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:16:19.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-13T15:16:19.850 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:16:19.850 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:16:19.850 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:16:19.850 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:16:20.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:16:13.050000+0000 '>' 2024-09-13T15:16:13.050000+0000 2024-09-13T15:16:20.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:16:21.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:16:21.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:16:21.183 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-13T15:16:21.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:16:21.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:16:21.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:16:21.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:16:21.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:16:13.050000+0000 '>' 2024-09-13T15:16:13.050000+0000 2024-09-13T15:16:21.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:16:22.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:16:22.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:16:22.502 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-13T15:16:22.502 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:16:22.502 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:16:22.503 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:16:22.503 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:16:22.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:16:19.088519+0000 '>' 2024-09-13T15:16:13.050000+0000 2024-09-13T15:16:22.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:16:22.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T15:16:24.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-09-13T15:16:24.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:16:24.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:16:24.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:16:24.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:16:24.824 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:16:24.824 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:16:24.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:16:24.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:16:24.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:16:24.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:16:24.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:16:24.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:16:24.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:16:24.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:16:24.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:16:25.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:16:25.291 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:16:25.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:16:25.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:16:25.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:16:25.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=867583393802 2024-09-13T15:16:25.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 867583393802 2024-09-13T15:16:25.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-867583393802' 2024-09-13T15:16:25.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:16:25.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:16:25.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=893353197575 2024-09-13T15:16:25.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 893353197575 2024-09-13T15:16:25.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-867583393802 1-893353197575' 2024-09-13T15:16:25.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:16:25.591 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-867583393802 2024-09-13T15:16:25.591 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:16:25.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:16:25.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-867583393802 2024-09-13T15:16:25.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:16:25.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=867583393802 2024-09-13T15:16:25.595 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 867583393802 2024-09-13T15:16:25.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 867583393802' 2024-09-13T15:16:25.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:16:26.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 867583393802 -lt 867583393802 2024-09-13T15:16:26.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:16:26.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:16:26.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-893353197575 2024-09-13T15:16:26.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:16:26.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-893353197575 2024-09-13T15:16:26.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:16:26.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=893353197575 2024-09-13T15:16:26.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 893353197575' 2024-09-13T15:16:26.027 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 893353197575 2024-09-13T15:16:26.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:16:26.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 893353197575 -lt 893353197575 2024-09-13T15:16:26.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:16:26.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:16:26.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:16:26.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:16:26.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:16:26.776 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:16:26.776 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:16:26.776 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:16:26.776 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:16:26.776 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:16:26.776 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:16:27.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:16:27.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:16:27.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:16:27.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:16:27.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:16:27.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:16:27.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:16:27.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-09-13T15:16:27.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-09-13T15:16:27.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-09-13T15:16:27.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-09-13T15:16:27.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-09-13T15:16:27.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-09-13T15:16:27.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-09-13T15:16:27.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-09-13T15:16:28.017 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'csr_pool' removed 2024-09-13T15:16:28.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T15:16:28.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:16:28.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:16:28.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:16:28.040 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:16:28.040 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:16:28.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:16:28.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:16:28.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:16:28.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:16:28.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:16:28.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:16:28.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:16:28.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:16:28.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:16:28.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:16:28.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:16:28.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:16:28.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:16:28.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:16:28.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:16:28.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:16:28.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:16:28.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:16:28.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:28.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:28.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:16:28.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:16:28.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:16:28.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T15:16:28.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T15:16:28.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T15:16:28.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T15:16:28.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:16:28.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:16:28.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:16:28.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:16:28.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:16:28.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:16:28.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:16:28.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:16:28.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:16:28.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:16:28.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:16:28.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:16:28.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:16:28.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:16:28.420 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:16:28.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:16:28.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:16:28.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:16:28.422 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:16:28.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:16:28.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:16:28.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:16:28.426 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:16:28.426 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:28.426 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:28.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:16:28.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:16:28.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:16:28.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T15:16:28.429 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:16:28.429 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:28.429 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:28.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T15:16:28.433 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:16:28.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T15:16:28.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:16:28.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T15:16:28.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-09-13T15:16:28.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5934: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-09-13T15:16:28.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-09-13T15:16:28.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5936: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-09-13T15:16:28.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5938: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-13T15:16:28.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T15:16:28.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:16:28.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:16:28.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:16:28.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T15:16:28.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-13T15:16:28.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:16:28.903 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:16:28.903 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:16:28.903 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:16:28.903 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:28.903 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:28.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:16:28.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-13T15:16:29.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:16:29.123 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:16:29.123 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:16:29.123 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:16:29.123 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:16:29.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:16:29.124 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:16:29.124 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:16:29.124 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:16:29.125 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:16:29.125 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:29.125 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:29.125 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:16:29.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:16:29.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T15:16:29.243 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:16:29.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:16:29.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:16:29.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:16:29.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:16:29.245 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:16:29.245 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:16:29.245 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:16:29.245 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:16:29.245 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:29.245 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:29.245 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:16:29.248 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:16:29.248 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T15:16:29.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5939: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-09-13T15:16:29.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T15:16:29.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:16:29.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:16:29.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:16:29.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T15:16:29.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:16:30.187 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:16:30.187 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:16:30.187 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:16:30.188 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:16:30.188 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:30.188 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:30.188 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:16:30.189 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:16:30.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:16:30.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5940: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-09-13T15:16:30.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:16:30.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:16:30.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:16:30.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:16:30.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:16:30.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:16:30.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:16:30.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:16:30.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:16:30.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:16:30.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:16:30.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:16:30.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:16:30.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:16:30.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:16:30.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:16:30.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:16:30.229 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:16:30.229 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:16:30.229 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:16:30.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:16:30.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:16:30.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:16:30.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:16:30.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:16:30.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:16:30.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:16:30.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:16:30.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:16:30.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:16:30.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:16:30.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:16:30.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:16:30.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:16:30.232 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:16:30.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1916e55e-ce79-43f3-aa3f-ce5dcd215a82 2024-09-13T15:16:30.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1916e55e-ce79-43f3-aa3f-ce5dcd215a82' 2024-09-13T15:16:30.232 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 1916e55e-ce79-43f3-aa3f-ce5dcd215a82 2024-09-13T15:16:30.233 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:16:30.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBOV+RmcqncDhAAsfBlbLH5uZ3/fdreq3TChg== 2024-09-13T15:16:30.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBOV+RmcqncDhAAsfBlbLH5uZ3/fdreq3TChg=="}' 2024-09-13T15:16:30.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1916e55e-ce79-43f3-aa3f-ce5dcd215a82 -i td/osd-scrub-repair/0/new.json 2024-09-13T15:16:30.508 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:16:30.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T15:16:30.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBOV+RmcqncDhAAsfBlbLH5uZ3/fdreq3TChg== --osd-uuid 1916e55e-ce79-43f3-aa3f-ce5dcd215a82 2024-09-13T15:16:30.559 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:16:30.552+0000 7fc5cd0b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:16:30.562 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:16:30.556+0000 7fc5cd0b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:16:30.563 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:16:30.556+0000 7fc5cd0b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:16:30.563 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:16:30.556+0000 7fc5cd0b77c0 -1 bdev(0x560b4d795c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:16:30.564 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:16:30.556+0000 7fc5cd0b77c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T15:16:36.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T15:16:36.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:16:36.746 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:16:36.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:16:36.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:16:37.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:16:37.163 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:16:37.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:16:37.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:16:37.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:16:37.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:16:37.196 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:16:37.196+0000 7faebdb297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:16:37.224 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:16:37.224+0000 7faebdb297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:16:37.225 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:16:37.224+0000 7faebdb297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:16:37.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:16:37.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:16:37.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:16:37.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:16:37.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:16:37.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:37.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:16:37.524 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:16:37.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:37.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:37.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:38.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:38.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:38.881 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:16:38.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:16:38.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:38.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:39.167 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:16:39.164+0000 7faebdb297c0 -1 Falling back to public interface 2024-09-13T15:16:39.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:40.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:40.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:40.199 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:16:40.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:16:40.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:40.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:40.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:41.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:41.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:41.519 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:16:41.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:16:41.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:41.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:41.822 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:16:41.820+0000 7faebdb297c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:16:41.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:42.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:42.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:42.853 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:16:42.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:16:42.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:42.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:43.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:44.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:44.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:44.187 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:16:44.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:16:44.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:44.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:44.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:45.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:45.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:45.590 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:16:45.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:16:45.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:45.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:45.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:46.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:46.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:46.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:16:46.923 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:16:46.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:46.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:47.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:48.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:48.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:48.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:16:48.251 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:16:48.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:48.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:48.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:49.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:49.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:49.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:16:49.583 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:16:49.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:49.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:49.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:50.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:50.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:50.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:16:50.909 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:16:50.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:50.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:51.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:52.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:52.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:52.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:16:52.247 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:16:52.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:52.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:52.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:53.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:53.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:53.605 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:16:53.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:16:53.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:53.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:54.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:55.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:55.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:55.277 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:16:55.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:16:55.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:55.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:56.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:57.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:57.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:57.142 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:16:57.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:16:57.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:57.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:57.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:16:58.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:16:58.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:16:58.861 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:16:58.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:16:58.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:16:58.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:16:59.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:00.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:00.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:00.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:17:00.667 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:17:00.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:00.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:17:01.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:02.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:02.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:02.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:17:02.315 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:17:02.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:02.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:17:03.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:04.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:04.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:04.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:17:04.004 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:17:04.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:04.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:17:04.352 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3822153151,v1:127.0.0.1:6803/3822153151] [v2:127.0.0.1:6804/3822153151,v1:127.0.0.1:6805/3822153151] exists,up 1916e55e-ce79-43f3-aa3f-ce5dcd215a82 2024-09-13T15:17:04.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:17:04.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:17:04.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:17:04.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5941: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-09-13T15:17:04.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:17:04.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:17:04.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T15:17:04.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:17:04.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:17:04.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:17:04.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:17:04.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:17:04.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:17:04.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:17:04.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:17:04.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:17:04.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:17:04.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:17:04.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:17:04.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:17:04.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:17:04.357 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:17:04.358 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:17:04.358 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:17:04.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:17:04.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:17:04.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:17:04.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:17:04.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:17:04.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:17:04.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:17:04.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:17:04.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:17:04.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:17:04.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:17:04.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:17:04.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:17:04.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:17:04.359 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:17:04.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4e346ce2-da21-405d-b907-156ba22d02e5 2024-09-13T15:17:04.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 4e346ce2-da21-405d-b907-156ba22d02e5' 2024-09-13T15:17:04.360 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 4e346ce2-da21-405d-b907-156ba22d02e5 2024-09-13T15:17:04.361 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:17:04.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBwV+RmlT/XFhAA3slQRsHu8Nbdv4cDMx09MA== 2024-09-13T15:17:04.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBwV+RmlT/XFhAA3slQRsHu8Nbdv4cDMx09MA=="}' 2024-09-13T15:17:04.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4e346ce2-da21-405d-b907-156ba22d02e5 -i td/osd-scrub-repair/1/new.json 2024-09-13T15:17:04.751 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:17:04.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T15:17:04.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBwV+RmlT/XFhAA3slQRsHu8Nbdv4cDMx09MA== --osd-uuid 4e346ce2-da21-405d-b907-156ba22d02e5 2024-09-13T15:17:04.804 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:17:04.803+0000 7fb3f65e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:17:04.827 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:17:04.823+0000 7fb3f65e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:17:04.828 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:17:04.827+0000 7fb3f65e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:17:04.828 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:17:04.827+0000 7fb3f65e77c0 -1 bdev(0x563a23007c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T15:17:04.828 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:17:04.827+0000 7fb3f65e77c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T15:17:10.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T15:17:10.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:17:10.898 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T15:17:10.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T15:17:10.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:17:11.322 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:17:11.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T15:17:11.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:17:11.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:17:11.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:17:11.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:17:11.355 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:17:11.351+0000 7f19a823b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:17:11.375 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:17:11.371+0000 7f19a823b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:17:11.376 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:17:11.375+0000 7f19a823b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:17:11.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T15:17:11.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:17:11.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:17:11.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:17:11.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:17:11.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:11.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:17:11.695 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:17:11.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:11.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:12.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:12.679 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:17:12.675+0000 7f19a823b7c0 -1 Falling back to public interface 2024-09-13T15:17:13.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:13.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:13.022 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:17:13.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:17:13.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:13.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:13.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:14.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:14.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:14.399 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:17:14.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:17:14.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:14.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:14.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:15.498 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:17:15.495+0000 7f19a823b7c0 -1 osd.1 0 log_to_monitors true 2024-09-13T15:17:15.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:15.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:15.743 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:17:15.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:17:15.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:15.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:16.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:17.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:17.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:17.184 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:17:17.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:17:17.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:17.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:17.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:18.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:18.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:18.598 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:17:18.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:17:18.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:18.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:18.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:19.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:19.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:19.948 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:17:19.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:17:19.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:19.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:20.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:21.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:21.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:21.402 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:17:21.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:17:21.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:21.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:21.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:22.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:22.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:22.813 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:17:22.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:17:22.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:22.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:23.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:24.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:24.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:24.159 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:17:24.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:17:24.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:24.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:24.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:25.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:25.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:25.485 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:17:25.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:17:25.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:25.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:25.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:26.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:26.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:26.823 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:17:26.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:17:26.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:26.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:27.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:28.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:28.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:28.249 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:17:28.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:17:28.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:28.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:28.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:29.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:29.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:29.876 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:17:29.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:17:29.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:29.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:30.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:31.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:31.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:31.617 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:17:31.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:17:31.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:31.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:32.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:33.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:33.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:33.453 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:17:33.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:17:33.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:33.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:34.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:35.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:35.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:35.220 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:17:35.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:17:35.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:35.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:35.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:36.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:36.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:36.766 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:17:36.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:17:36.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:36.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:37.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:38.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:38.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:38.328 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:17:38.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:17:38.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:38.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:38.728 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4090870506,v1:127.0.0.1:6811/4090870506] [v2:127.0.0.1:6812/4090870506,v1:127.0.0.1:6813/4090870506] exists,up 4e346ce2-da21-405d-b907-156ba22d02e5 2024-09-13T15:17:38.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:17:38.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:17:38.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:17:38.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-09-13T15:17:38.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T15:17:39.059 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' does not exist 2024-09-13T15:17:39.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T15:17:39.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T15:17:39.586 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T15:17:39.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:17:40.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T15:17:42.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-09-13T15:17:42.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:17:42.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:17:42.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:17:42.065 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:17:42.066 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:17:42.066 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:17:42.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:17:42.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:17:42.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:17:42.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:17:42.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:17:42.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:17:42.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:17:42.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:17:42.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:17:42.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:17:42.534 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:17:42.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:17:42.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:17:42.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:17:42.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836490 2024-09-13T15:17:42.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836490 2024-09-13T15:17:42.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490' 2024-09-13T15:17:42.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:17:42.697 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:17:42.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672969 2024-09-13T15:17:42.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672969 2024-09-13T15:17:42.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2024-09-13T15:17:42.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:17:42.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836490 2024-09-13T15:17:42.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:17:42.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:17:42.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836490 2024-09-13T15:17:42.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:17:42.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836490 2024-09-13T15:17:42.844 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836490 2024-09-13T15:17:42.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-13T15:17:42.845 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:17:43.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836490 2024-09-13T15:17:43.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:17:43.165 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672969 2024-09-13T15:17:43.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:17:43.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:17:43.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672969 2024-09-13T15:17:43.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:17:43.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672969 2024-09-13T15:17:43.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-13T15:17:43.172 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672969 2024-09-13T15:17:43.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:17:43.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672969 -lt 42949672969 2024-09-13T15:17:43.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:17:43.496 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:17:43.497 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:17:43.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T15:17:43.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:17:43.864 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:17:43.864 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:17:43.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:17:43.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:17:43.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:17:43.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:17:44.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T15:17:44.198 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:17:44.199 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:17:44.199 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:17:44.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T15:17:44.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:17:44.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:17:44.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5945: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-09-13T15:17:44.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2024-09-13T15:17:44.981 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'foo' created 2024-09-13T15:17:45.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:17:46.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-09-13T15:17:46.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2024-09-13T15:17:46.431 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'csr_pool' created 2024-09-13T15:17:46.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:17:47.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-09-13T15:17:47.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:17:47.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:17:47.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:17:47.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:17:47.453 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:17:47.453 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:17:47.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:17:47.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:17:47.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:17:47.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:17:47.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:17:47.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:17:47.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:17:47.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:17:47.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:17:47.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:17:47.933 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:17:47.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:17:47.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:17:47.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:17:48.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836492 2024-09-13T15:17:48.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836492 2024-09-13T15:17:48.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836492' 2024-09-13T15:17:48.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:17:48.102 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:17:48.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672971 2024-09-13T15:17:48.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672971 2024-09-13T15:17:48.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836492 1-42949672971' 2024-09-13T15:17:48.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:17:48.241 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836492 2024-09-13T15:17:48.241 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:17:48.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:17:48.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836492 2024-09-13T15:17:48.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:17:48.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836492 2024-09-13T15:17:48.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-13T15:17:48.245 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836492 2024-09-13T15:17:48.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:17:48.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836491 -lt 21474836492 2024-09-13T15:17:48.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:17:49.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:17:49.569 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:17:49.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-13T15:17:49.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:17:49.892 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672971 2024-09-13T15:17:49.892 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:17:49.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:17:49.895 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672971 2024-09-13T15:17:49.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:17:49.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672971 2024-09-13T15:17:49.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-09-13T15:17:49.897 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672971 2024-09-13T15:17:49.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:17:50.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672971 -lt 42949672971 2024-09-13T15:17:50.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:17:50.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:17:50.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:17:50.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:17:50.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:17:50.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:17:50.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:17:50.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:17:50.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:17:50.636 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:17:50.636 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:17:50.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:17:50.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:17:50.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:17:50.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:17:51.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:17:51.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:17:51.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:17:51.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5949: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-09-13T15:17:51.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5949: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-13T15:17:51.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-09-13T15:17:51.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-09-13T15:17:51.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:17:51.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:17:51.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-09-13T15:17:51.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:17:51.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:17:51.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:17:51.744 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:17:51.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:17:52.280 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:17:52.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:17:52.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:17:52.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:17:52.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5953: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-09-13T15:17:52.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-09-13T15:17:52.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5949: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-13T15:17:52.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-09-13T15:17:52.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-09-13T15:17:52.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:17:52.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-13T15:17:52.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-09-13T15:17:52.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:17:52.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:17:52.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:17:53.216 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:17:53.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:17:53.752 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:17:53.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:17:53.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:17:53.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:17:53.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5953: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-09-13T15:17:54.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-09-13T15:17:54.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-09-13T15:17:54.243 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2024-09-13T15:17:54.243 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2024-09-13T15:17:54.243 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-09-13T15:17:54.243 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T15:17:54.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-09-13T15:17:54.555 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-09-13T15:17:54.556 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2024-09-13T15:17:54.556 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2024-09-13T15:17:54.556 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-09-13T15:17:54.556 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:17:54.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-09-13T15:17:54.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-09-13T15:17:55.143 INFO:tasks.workunit.client.0.smithi112.stdout:created pool csr_pool snap snap1 2024-09-13T15:17:55.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-09-13T15:17:55.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5963: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-09-13T15:17:55.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5963: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-13T15:17:55.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-09-13T15:17:55.149 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-09-13T15:17:55.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-09-13T15:17:55.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: case $i in 2024-09-13T15:17:55.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5971: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-09-13T15:17:55.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5972: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-09-13T15:17:55.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:17:55.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:17:55.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T15:17:55.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:17:55.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-09-13T15:17:55.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:17:55.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:17:55.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T15:17:55.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:17:55.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:17:55.397 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:17:55.397 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:17:55.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:17:55.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:17:55.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:17:55.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:17:55.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-09-13T15:17:55.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:17:55.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:17:55.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:17:55.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:17:55.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:17:55.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-09-13T15:17:58.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-13T15:17:58.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:17:58.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:17:58.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:17:58.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:17:58.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:17:58.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:17:58.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:17:58.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:17:58.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:17:58.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:17:58.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:17:58.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:17:58.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:17:58.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:17:58.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:17:58.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:17:58.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:17:58.129 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:17:58.130 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:17:58.130 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:17:58.130 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:17:58.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:17:58.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:17:58.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:17:58.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:17:58.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:17:58.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:17:58.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:17:58.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:17:58.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:17:58.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:17:58.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:17:58.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:17:58.133 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:17:58.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:17:58.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:17:58.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:17:58.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:17:58.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:17:58.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:17:58.166 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:17:58.162+0000 7fafcd6597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:17:58.169 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:17:58.166+0000 7fafcd6597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:17:58.171 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:17:58.166+0000 7fafcd6597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:17:58.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:17:58.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:17:58.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:17:58.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:17:58.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:17:58.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:58.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:17:58.473 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:17:58.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:17:58.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:58.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:17:59.666 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:17:59.662+0000 7fafcd6597c0 -1 Falling back to public interface 2024-09-13T15:17:59.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:17:59.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:17:59.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:17:59.812 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:17:59.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:17:59.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:18:00.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:01.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:01.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:01.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:18:01.141 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:18:01.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:01.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:18:01.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:02.362 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:02.358+0000 7fafcd6597c0 -1 osd.1 28 log_to_monitors true 2024-09-13T15:18:02.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:02.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:02.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:18:02.482 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:18:02.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:02.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:18:02.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:03.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:03.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:03.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:18:03.869 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:18:03.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:03.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:18:04.212 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/3724197909,v1:127.0.0.1:6811/3724197909] [v2:127.0.0.1:6812/3724197909,v1:127.0.0.1:6813/3724197909] exists,up 4e346ce2-da21-405d-b907-156ba22d02e5 2024-09-13T15:18:04.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:18:04.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:18:04.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:18:04.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:18:04.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:18:04.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:18:04.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:18:04.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:18:04.214 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:18:04.214 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:18:04.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:18:04.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:18:04.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:18:04.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:18:04.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:18:04.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:18:04.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:18:04.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:18:04.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:18:04.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:18:04.665 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:18:04.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:18:04.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:18:04.665 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:18:04.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836497 2024-09-13T15:18:04.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836497 2024-09-13T15:18:04.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836497' 2024-09-13T15:18:04.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:18:04.836 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:18:04.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953474 2024-09-13T15:18:04.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953474 2024-09-13T15:18:04.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836497 1-137438953474' 2024-09-13T15:18:04.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:18:04.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836497 2024-09-13T15:18:04.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:18:04.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:18:04.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836497 2024-09-13T15:18:04.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:18:04.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836497 2024-09-13T15:18:04.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-09-13T15:18:04.981 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836497 2024-09-13T15:18:04.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:18:05.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836497 2024-09-13T15:18:05.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:18:05.285 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-137438953474 2024-09-13T15:18:05.286 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:18:05.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:18:05.289 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-137438953474 2024-09-13T15:18:05.289 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:18:05.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953474 2024-09-13T15:18:05.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-09-13T15:18:05.291 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 137438953474 2024-09-13T15:18:05.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:18:05.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 137438953474 2024-09-13T15:18:05.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:18:06.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:18:06.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:18:06.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953474 -lt 137438953474 2024-09-13T15:18:06.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:18:06.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:18:06.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:18:07.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:18:07.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:18:07.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:18:07.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:18:07.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:18:07.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:18:07.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:18:07.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:18:07.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:18:07.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:18:07.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:18:07.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:18:08.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:18:08.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:18:08.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:18:08.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5963: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-09-13T15:18:08.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-09-13T15:18:08.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-09-13T15:18:08.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5967: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-09-13T15:18:08.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: case $i in 2024-09-13T15:18:08.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5976: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-09-13T15:18:08.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5977: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-09-13T15:18:08.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:18:08.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:18:08.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:18:08.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:18:08.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-09-13T15:18:08.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:18:08.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:18:08.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:18:08.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:18:08.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:18:08.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:18:08.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:18:08.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:18:08.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:18:08.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:18:08.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:18:08.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-09-13T15:18:08.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:18:08.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:18:08.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:18:08.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:18:08.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:18:08.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-09-13T15:18:10.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:18:10.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:18:10.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:18:10.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:18:10.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:18:10.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:18:10.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:18:10.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:18:10.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:18:10.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:18:10.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:18:10.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:18:10.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:18:10.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:18:10.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:18:10.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:18:10.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:18:10.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:18:10.724 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:18:10.724 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:18:10.724 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:18:10.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:18:10.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:18:10.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:18:10.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:18:10.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:18:10.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:18:10.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:18:10.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:18:10.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:18:10.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:18:10.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:18:10.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:18:10.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:18:10.730 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:18:10.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:18:10.731 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:18:10.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:18:10.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:18:10.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:18:10.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:18:10.758 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:10.754+0000 7f0482c657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:18:10.761 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:10.758+0000 7f0482c657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:18:10.763 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:10.758+0000 7f0482c657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:18:11.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:18:11.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:18:11.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:18:11.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:18:11.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:18:11.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:11.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:18:11.060 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:18:11.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:11.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:11.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:12.219 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:12.214+0000 7f0482c657c0 -1 Falling back to public interface 2024-09-13T15:18:12.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:12.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:12.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:18:12.437 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:18:12.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:12.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:12.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:13.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:13.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:13.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:18:13.772 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:18:13.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:13.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:14.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:15.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:15.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:15.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:18:15.113 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:18:15.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:15.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:15.161 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:15.158+0000 7f0482c657c0 -1 osd.0 34 log_to_monitors true 2024-09-13T15:18:15.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:16.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:16.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:16.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:18:16.474 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:18:16.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:16.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:16.810 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [5,34) [v2:127.0.0.1:6802/1404923321,v1:127.0.0.1:6803/1404923321] [v2:127.0.0.1:6804/1404923321,v1:127.0.0.1:6805/1404923321] exists,up 1916e55e-ce79-43f3-aa3f-ce5dcd215a82 2024-09-13T15:18:16.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:18:16.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:18:16.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:18:16.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:18:16.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:18:16.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:18:16.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:18:16.811 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:18:16.812 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:18:16.812 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:18:16.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:18:16.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:18:16.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:18:16.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:18:16.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:18:16.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:18:16.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:18:16.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:18:16.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:18:17.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:18:17.261 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:18:17.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:18:17.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:18:17.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:18:17.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757250 2024-09-13T15:18:17.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757250 2024-09-13T15:18:17.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757250' 2024-09-13T15:18:17.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:18:17.438 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:18:17.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953477 2024-09-13T15:18:17.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953477 2024-09-13T15:18:17.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757250 1-137438953477' 2024-09-13T15:18:17.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:18:17.583 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-163208757250 2024-09-13T15:18:17.583 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:18:17.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:18:17.584 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-163208757250 2024-09-13T15:18:17.584 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:18:17.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757250 2024-09-13T15:18:17.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 163208757250' 2024-09-13T15:18:17.586 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 163208757250 2024-09-13T15:18:17.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:18:17.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 163208757250 2024-09-13T15:18:17.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:18:18.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:18:18.908 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:18:19.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757250 -lt 163208757250 2024-09-13T15:18:19.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:18:19.286 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-137438953477 2024-09-13T15:18:19.286 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:18:19.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:18:19.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-137438953477 2024-09-13T15:18:19.289 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:18:19.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953477 2024-09-13T15:18:19.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 137438953477' 2024-09-13T15:18:19.290 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 137438953477 2024-09-13T15:18:19.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:18:19.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953478 -lt 137438953477 2024-09-13T15:18:19.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:18:19.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:18:19.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:18:20.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:18:20.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:18:20.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:18:20.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:18:20.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:18:20.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:18:20.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:18:20.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:18:20.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:18:20.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:18:20.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:18:20.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:18:20.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:18:20.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:18:20.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:18:20.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5982: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-09-13T15:18:20.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5984: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-09-13T15:18:20.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=3.0 2024-09-13T15:18:20.702 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 3.0 2024-09-13T15:18:20.703 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:18:20.703 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:18:20.703 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:18:20.703 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:18:21.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-13T15:17:46.385865+0000 2024-09-13T15:18:21.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 3.0 2024-09-13T15:18:21.310 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 3.0 on osd.1 to scrub 2024-09-13T15:18:21.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 3.0 2024-09-13T15:17:46.385865+0000 2024-09-13T15:18:21.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-13T15:18:21.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:17:46.385865+0000 2024-09-13T15:18:21.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:18:21.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:18:21.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:18:21.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-13T15:18:21.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:18:21.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:18:21.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:18:21.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:18:21.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:17:46.385865+0000 '>' 2024-09-13T15:17:46.385865+0000 2024-09-13T15:18:21.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:18:22.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:18:22.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:18:22.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-13T15:18:22.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-13T15:18:22.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:18:22.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:18:22.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-13T15:18:22.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:18:21.466550+0000 '>' 2024-09-13T15:17:46.385865+0000 2024-09-13T15:18:22.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:18:22.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5986: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-09-13T15:18:23.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5988: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-09-13T15:18:23.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5988: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-09-13T15:18:23.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5990: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-13T15:18:23.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5990: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-09-13T15:18:23.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5992: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-09-13T15:18:23.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5994: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-09-13T15:18:23.107 INFO:tasks.workunit.client.0.smithi112.stderr: . as $in 2024-09-13T15:18:23.107 INFO:tasks.workunit.client.0.smithi112.stderr: | if type == "object" then 2024-09-13T15:18:23.107 INFO:tasks.workunit.client.0.smithi112.stderr: reduce keys[] as $key 2024-09-13T15:18:23.107 INFO:tasks.workunit.client.0.smithi112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-13T15:18:23.107 INFO:tasks.workunit.client.0.smithi112.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-13T15:18:23.108 INFO:tasks.workunit.client.0.smithi112.stderr: else f 2024-09-13T15:18:23.108 INFO:tasks.workunit.client.0.smithi112.stderr: end; 2024-09-13T15:18:23.108 INFO:tasks.workunit.client.0.smithi112.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-13T15:18:23.108 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-13T15:18:23.108 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-13T15:18:23.108 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-13T15:18:23.108 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-13T15:18:23.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5994: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-09-13T15:18:23.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5994: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-13T15:18:23.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-09-13T15:18:23.144 INFO:tasks.workunit.client.0.smithi112.stderr: . as $in 2024-09-13T15:18:23.144 INFO:tasks.workunit.client.0.smithi112.stderr: | if type == "object" then 2024-09-13T15:18:23.144 INFO:tasks.workunit.client.0.smithi112.stderr: reduce keys[] as $key 2024-09-13T15:18:23.144 INFO:tasks.workunit.client.0.smithi112.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-13T15:18:23.144 INFO:tasks.workunit.client.0.smithi112.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-13T15:18:23.144 INFO:tasks.workunit.client.0.smithi112.stderr: else f 2024-09-13T15:18:23.144 INFO:tasks.workunit.client.0.smithi112.stderr: end; 2024-09-13T15:18:23.145 INFO:tasks.workunit.client.0.smithi112.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-13T15:18:23.145 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-13T15:18:23.145 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-13T15:18:23.145 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-13T15:18:23.145 INFO:tasks.workunit.client.0.smithi112.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-13T15:18:23.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-09-13T15:18:23.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-13T15:18:23.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6161: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-13T15:18:23.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-13T15:18:23.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6162: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-09-13T15:18:23.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6167: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-09-13T15:18:23.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6172: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-09-13T15:18:23.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6173: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-09-13T15:18:23.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6174: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-09-13T15:18:23.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6175: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-09-13T15:18:23.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-09-13T15:18:23.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6177: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-09-13T15:18:23.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-09-13T15:18:23.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6179: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-09-13T15:18:23.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-13T15:18:23.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-09-13T15:18:23.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-13T15:18:23.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-09-13T15:18:23.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-13T15:18:23.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-09-13T15:18:23.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-13T15:18:23.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-09-13T15:18:23.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-13T15:18:23.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-09-13T15:18:23.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6181: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-09-13T15:18:23.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6183: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-09-13T15:18:23.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6190: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-09-13T15:18:23.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6196: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-09-13T15:18:23.599 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'csr_pool' removed 2024-09-13T15:18:23.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T15:18:23.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:18:23.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:18:23.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:18:23.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:18:23.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:18:23.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:18:23.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:18:23.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:18:23.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:18:23.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:18:23.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:18:23.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:18:23.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:18:23.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:18:23.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:18:23.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:18:23.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:18:23.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:18:23.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:18:23.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:18:23.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:18:23.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:18:23.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:18:23.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:18:23.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:18:23.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:18:23.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:18:23.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:18:23.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T15:18:23.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T15:18:23.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T15:18:23.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T15:18:23.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:18:23.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:18:23.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:18:23.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:18:23.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:18:23.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:18:23.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:18:23.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:18:23.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:18:23.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:18:23.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:18:23.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:18:23.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:18:23.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:18:23.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:18:23.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:18:23.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:18:23.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:18:23.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:18:23.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:18:23.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:18:23.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:18:23.976 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:18:23.976 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:18:23.976 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:18:23.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:18:23.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:18:23.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:18:23.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T15:18:23.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:18:23.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:18:23.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:18:23.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T15:18:23.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:18:23.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T15:18:23.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:18:23.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T15:18:23.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-09-13T15:18:23.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-09-13T15:18:23.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-09-13T15:18:23.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=false 2024-09-13T15:18:23.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-09-13T15:18:23.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-13T15:18:23.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T15:18:23.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:18:23.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:18:23.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:18:23.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T15:18:23.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T15:18:24.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:18:24.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:18:24.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:18:24.453 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:18:24.453 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:18:24.453 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:18:24.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:18:24.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T15:18:24.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:18:24.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:18:24.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:18:24.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:18:24.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:18:24.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:18:24.663 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:18:24.663 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:18:24.663 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:18:24.664 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:18:24.664 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:18:24.664 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:18:24.664 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:18:24.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:18:24.665 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T15:18:24.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:18:24.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:18:24.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:18:24.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:18:24.778 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:18:24.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:18:24.778 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:18:24.778 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:18:24.779 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:18:24.779 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:18:24.779 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:18:24.779 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:18:24.779 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:18:24.779 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T15:18:24.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-13T15:18:24.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T15:18:24.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:18:24.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:18:24.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:18:24.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T15:18:24.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:18:25.745 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:18:25.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:18:25.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:18:25.748 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:18:25.748 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:18:25.748 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:18:25.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:18:25.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:18:25.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:18:25.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-09-13T15:18:25.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:18:25.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-09-13T15:18:25.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:18:25.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:18:25.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:18:25.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:18:25.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:18:25.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:18:25.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:18:25.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:18:25.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:18:25.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:18:25.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:18:25.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:18:25.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:18:25.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:18:25.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:18:25.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:18:25.788 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:18:25.788 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:18:25.788 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:18:25.788 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:18:25.788 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:18:25.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:18:25.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:18:25.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:18:25.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:18:25.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:18:25.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:18:25.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:18:25.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:18:25.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:18:25.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:18:25.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:18:25.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:18:25.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:18:25.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:18:25.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2a8f688e-3e1d-4070-9780-dbb2b2a8ea34 2024-09-13T15:18:25.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2a8f688e-3e1d-4070-9780-dbb2b2a8ea34' 2024-09-13T15:18:25.791 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 2a8f688e-3e1d-4070-9780-dbb2b2a8ea34 2024-09-13T15:18:25.791 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:18:25.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDBV+RmxNsqMBAA7DkMbRzAiR4PBkN3YUlGSw== 2024-09-13T15:18:25.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDBV+RmxNsqMBAA7DkMbRzAiR4PBkN3YUlGSw=="}' 2024-09-13T15:18:25.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2a8f688e-3e1d-4070-9780-dbb2b2a8ea34 -i td/osd-scrub-repair/0/new.json 2024-09-13T15:18:26.049 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:18:26.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T15:18:26.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDBV+RmxNsqMBAA7DkMbRzAiR4PBkN3YUlGSw== --osd-uuid 2a8f688e-3e1d-4070-9780-dbb2b2a8ea34 2024-09-13T15:18:26.095 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:26.089+0000 7fed92a857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:18:26.099 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:26.093+0000 7fed92a857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:18:26.101 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:26.097+0000 7fed92a857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:18:26.102 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:26.097+0000 7fed92a857c0 -1 bdev(0x56412e1c3c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:18:26.102 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:26.097+0000 7fed92a857c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T15:18:32.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T15:18:32.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:18:32.455 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:18:32.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:18:32.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:18:32.872 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:18:32.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:18:32.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:18:32.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:18:32.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:18:32.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:18:32.906 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:32.901+0000 7f38212777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:18:32.929 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:32.925+0000 7f38212777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:18:32.931 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:32.925+0000 7f38212777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:18:33.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:18:33.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:18:33.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:18:33.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:18:33.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:18:33.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:33.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:18:33.195 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:18:33.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:33.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:33.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:34.250 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:34.245+0000 7f38212777c0 -1 Falling back to public interface 2024-09-13T15:18:34.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:34.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:34.541 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:18:34.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:18:34.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:34.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:34.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:35.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:35.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:35.881 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:18:35.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:18:35.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:35.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:36.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:36.927 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:18:36.921+0000 7f38212777c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:18:37.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:37.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:37.217 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:18:37.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:18:37.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:37.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:37.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:38.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:38.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:38.557 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:18:38.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:18:38.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:38.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:38.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:39.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:39.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:39.894 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:18:39.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:18:39.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:39.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:40.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:41.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:41.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:41.226 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:18:41.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:18:41.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:41.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:41.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:42.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:42.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:42.624 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:18:42.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:18:42.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:42.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:42.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:43.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:43.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:43.962 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:18:43.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:18:43.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:43.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:44.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:45.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:45.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:45.288 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:18:45.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:18:45.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:45.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:45.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:46.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:46.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:46.664 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:18:46.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:18:46.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:46.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:46.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:48.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:48.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:48.001 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:18:48.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:18:48.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:48.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:48.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:49.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:49.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:49.333 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:18:49.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:18:49.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:49.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:50.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:51.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:51.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:51.013 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:18:51.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:18:51.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:51.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:51.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:52.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:52.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:52.917 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:18:52.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:18:52.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:52.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:53.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:54.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:54.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:54.912 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:18:54.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:18:54.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:54.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:55.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:56.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:56.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:56.436 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:18:56.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:18:56.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:56.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:56.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:57.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:57.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:57.824 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:18:57.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:18:57.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:57.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:58.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:18:59.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:18:59.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:18:59.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:18:59.605 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:18:59.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:18:59.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:18:59.935 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/10632230,v1:127.0.0.1:6803/10632230] [v2:127.0.0.1:6804/10632230,v1:127.0.0.1:6805/10632230] exists,up 2a8f688e-3e1d-4070-9780-dbb2b2a8ea34 2024-09-13T15:18:59.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:18:59.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:18:59.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:18:59.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:18:59.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-09-13T15:18:59.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:18:59.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:18:59.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T15:18:59.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:18:59.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:18:59.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:18:59.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:18:59.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:18:59.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:18:59.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:18:59.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:18:59.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:18:59.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:18:59.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:18:59.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:18:59.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:18:59.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:18:59.940 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:18:59.940 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:18:59.940 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:18:59.941 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:18:59.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:18:59.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:18:59.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:18:59.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:18:59.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:18:59.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:18:59.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:18:59.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:18:59.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:18:59.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:18:59.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:18:59.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:18:59.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:18:59.942 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:18:59.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b022a19b-2276-4df6-a43b-d9d2f107f669 2024-09-13T15:18:59.942 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 b022a19b-2276-4df6-a43b-d9d2f107f669 2024-09-13T15:18:59.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 b022a19b-2276-4df6-a43b-d9d2f107f669' 2024-09-13T15:18:59.943 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:18:59.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDjV+RmEntvORAAj6MidE0mhQWFfqbGznYVbg== 2024-09-13T15:18:59.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDjV+RmEntvORAAj6MidE0mhQWFfqbGznYVbg=="}' 2024-09-13T15:18:59.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b022a19b-2276-4df6-a43b-d9d2f107f669 -i td/osd-scrub-repair/1/new.json 2024-09-13T15:19:00.428 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:19:00.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T15:19:00.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDjV+RmEntvORAAj6MidE0mhQWFfqbGznYVbg== --osd-uuid b022a19b-2276-4df6-a43b-d9d2f107f669 2024-09-13T15:19:00.488 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:00.484+0000 7f759af0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:19:00.516 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:00.512+0000 7f759af0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:19:00.518 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:00.512+0000 7f759af0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:19:00.518 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:00.512+0000 7f759af0c7c0 -1 bdev(0x5600480a1c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T15:19:00.518 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:00.512+0000 7f759af0c7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T15:19:06.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T15:19:06.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:19:06.356 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T15:19:06.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T15:19:06.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:19:06.765 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:19:06.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T15:19:06.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:19:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:19:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:19:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:19:06.796 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:06.792+0000 7fe54a2bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:19:06.798 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:06.792+0000 7fe54a2bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:19:06.800 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:06.796+0000 7fe54a2bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:19:07.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T15:19:07.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:19:07.109 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:19:07.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:19:07.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:19:07.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:19:07.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:07.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:19:07.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:07.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:07.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:08.331 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:08.328+0000 7fe54a2bb7c0 -1 Falling back to public interface 2024-09-13T15:19:08.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:08.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:08.436 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:19:08.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:19:08.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:08.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:08.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:09.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:09.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:09.775 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:19:09.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:19:09.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:09.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:10.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:11.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:11.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:11.101 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:19:11.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:19:11.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:11.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:11.147 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:11.144+0000 7fe54a2bb7c0 -1 osd.1 0 log_to_monitors true 2024-09-13T15:19:11.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:12.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:12.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:12.470 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:19:12.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:19:12.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:12.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:12.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:13.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:13.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:13.795 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:19:13.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:19:13.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:13.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:14.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:15.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:15.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:15.202 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:19:15.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:19:15.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:15.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:15.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:16.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:16.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:16.578 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:19:16.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:19:16.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:16.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:17.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:18.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:18.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:18.003 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:19:18.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:19:18.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:18.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:18.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:19.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:19.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:19.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:19:19.343 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:19:19.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:19.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:19.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:20.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:20.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:20.684 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:19:20.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:19:20.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:20.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:21.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:22.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:22.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:22.083 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:19:22.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:19:22.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:22.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:22.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:23.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:23.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:23.421 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:19:23.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:19:23.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:23.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:24.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:25.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:25.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:25.074 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:19:25.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:19:25.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:25.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:25.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:26.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:26.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:26.709 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:19:26.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:19:26.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:26.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:27.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:28.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:28.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:28.102 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:19:28.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:19:28.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:28.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:29.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:30.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:30.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:30.120 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:19:30.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:19:30.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:30.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:30.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:31.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:31.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:31.700 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:19:31.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:19:31.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:31.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:32.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:33.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:33.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:33.294 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:19:33.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:19:33.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:33.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:33.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:34.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:34.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:34.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:19:34.774 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:19:34.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:34.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:19:35.108 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3169998765,v1:127.0.0.1:6811/3169998765] [v2:127.0.0.1:6812/3169998765,v1:127.0.0.1:6813/3169998765] exists,up b022a19b-2276-4df6-a43b-d9d2f107f669 2024-09-13T15:19:35.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:19:35.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:19:35.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:19:35.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:19:35.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-09-13T15:19:35.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:19:35.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:19:35.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T15:19:35.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:19:35.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:19:35.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:19:35.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:19:35.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:19:35.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:19:35.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T15:19:35.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T15:19:35.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:19:35.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:19:35.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:19:35.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:19:35.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:19:35.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:19:35.112 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:19:35.113 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:19:35.113 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:19:35.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:19:35.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:19:35.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:19:35.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:19:35.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:19:35.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:19:35.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:19:35.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:19:35.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:19:35.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:19:35.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:19:35.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:19:35.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:19:35.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T15:19:35.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:19:35.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=12a91fc7-df44-4dff-a272-14e32ffbd180 2024-09-13T15:19:35.114 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 12a91fc7-df44-4dff-a272-14e32ffbd180 2024-09-13T15:19:35.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 12a91fc7-df44-4dff-a272-14e32ffbd180' 2024-09-13T15:19:35.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:19:35.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAHWORmnJodCBAAy49f8/at4qf3GTC07+zNNg== 2024-09-13T15:19:35.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAHWORmnJodCBAAy49f8/at4qf3GTC07+zNNg=="}' 2024-09-13T15:19:35.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 12a91fc7-df44-4dff-a272-14e32ffbd180 -i td/osd-scrub-repair/2/new.json 2024-09-13T15:19:35.508 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:19:35.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T15:19:35.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHWORmnJodCBAAy49f8/at4qf3GTC07+zNNg== --osd-uuid 12a91fc7-df44-4dff-a272-14e32ffbd180 2024-09-13T15:19:35.564 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:35.563+0000 7f16824f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:19:35.596 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:35.595+0000 7f16824f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:19:35.597 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:35.595+0000 7f16824f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:19:35.598 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:35.595+0000 7f16824f37c0 -1 bdev(0x55cdc7327c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T15:19:35.598 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:35.595+0000 7f16824f37c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T15:19:42.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T15:19:42.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:19:42.244 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T15:19:42.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T15:19:42.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:19:42.656 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T15:19:42.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T15:19:42.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:19:42.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:19:42.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:19:42.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:19:42.690 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:42.687+0000 7f1e3bc357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:19:42.704 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:42.703+0000 7f1e3bc357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:19:42.705 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:42.703+0000 7f1e3bc357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:19:42.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T15:19:42.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:19:42.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:19:42.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:19:42.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:19:42.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:42.980 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:19:42.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:19:42.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:42.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:19:43.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:43.750 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:43.747+0000 7f1e3bc357c0 -1 Falling back to public interface 2024-09-13T15:19:44.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:44.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:44.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:19:44.306 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:19:44.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:44.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:19:44.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:45.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:45.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:45.624 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:19:45.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:19:45.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:45.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:19:45.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:46.694 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:19:46.691+0000 7f1e3bc357c0 -1 osd.2 0 log_to_monitors true 2024-09-13T15:19:46.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:46.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:46.957 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:19:46.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:19:46.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:46.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:19:47.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:48.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:48.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:48.281 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:19:48.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:19:48.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:48.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:19:48.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:49.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:49.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:49.626 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:19:49.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:19:49.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:49.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:19:49.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:50.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:50.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:50.965 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:19:50.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:19:50.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:50.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:19:51.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:52.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:52.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:52.390 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:19:52.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:19:52.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:52.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:19:52.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:53.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:53.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:53.750 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:19:53.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:19:53.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:53.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:19:54.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:55.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:55.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:55.085 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:19:55.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:19:55.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:55.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:19:55.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:56.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:56.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:56.427 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:19:56.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:19:56.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:56.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:19:56.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:57.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:57.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:57.800 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:19:57.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:19:57.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:57.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:19:58.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:19:59.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:19:59.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:19:59.137 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:19:59.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:19:59.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:19:59.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:19:59.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:00.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:00.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:00.656 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:20:00.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:20:00.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:00.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:20:01.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:02.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:02.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:02.391 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:20:02.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:20:02.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:02.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:20:03.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:04.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:04.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:04.148 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:20:04.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:20:04.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:04.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:20:04.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:05.875 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:20:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:20:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:20:06.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:07.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:07.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:07.669 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:20:07.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:20:07.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:07.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:20:08.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:09.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:09.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:09.095 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:20:09.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:20:09.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:09.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:20:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:10.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:10.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:10.520 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:20:10.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:20:10.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:10.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:20:10.865 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/706172654,v1:127.0.0.1:6819/706172654] [v2:127.0.0.1:6820/706172654,v1:127.0.0.1:6821/706172654] exists,up 12a91fc7-df44-4dff-a272-14e32ffbd180 2024-09-13T15:20:10.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:20:10.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:20:10.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:20:10.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-09-13T15:20:10.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T15:20:11.184 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' does not exist 2024-09-13T15:20:11.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T15:20:11.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T15:20:11.637 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T15:20:11.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:20:12.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T15:20:14.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-09-13T15:20:14.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:20:14.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:20:14.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:20:14.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:20:14.998 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:20:14.998 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:20:14.999 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:20:14.999 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:20:14.999 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:20:15.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:20:15.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:20:15.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:20:15.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:20:15.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:20:15.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:20:15.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:20:15.471 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:20:15.471 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:20:15.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:20:15.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:20:15.471 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:20:15.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836500 2024-09-13T15:20:15.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836500 2024-09-13T15:20:15.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500' 2024-09-13T15:20:15.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:20:15.628 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:20:15.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672971 2024-09-13T15:20:15.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672971 2024-09-13T15:20:15.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672971' 2024-09-13T15:20:15.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:20:15.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:20:15.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509446 2024-09-13T15:20:15.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509446 2024-09-13T15:20:15.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-42949672971 2-64424509446' 2024-09-13T15:20:15.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:20:15.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:20:15.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836500 2024-09-13T15:20:15.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:20:15.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836500 2024-09-13T15:20:15.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:20:15.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836500 2024-09-13T15:20:15.918 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836500 2024-09-13T15:20:15.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-13T15:20:15.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:20:16.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836500 2024-09-13T15:20:16.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:20:17.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:20:17.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:20:17.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-13T15:20:17.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:20:17.562 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:20:17.562 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672971 2024-09-13T15:20:17.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:20:17.564 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672971 2024-09-13T15:20:17.564 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:20:17.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672971 2024-09-13T15:20:17.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-09-13T15:20:17.565 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672971 2024-09-13T15:20:17.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:20:17.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672971 -lt 42949672971 2024-09-13T15:20:17.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:20:17.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509446 2024-09-13T15:20:17.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:20:17.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:20:17.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509446 2024-09-13T15:20:17.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:20:17.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509446 2024-09-13T15:20:17.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-13T15:20:17.906 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509446 2024-09-13T15:20:17.906 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:20:18.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-13T15:20:18.233 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:20:18.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:20:18.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:20:18.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T15:20:18.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:20:18.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:20:18.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:20:18.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:20:18.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:20:18.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:20:18.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:20:18.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T15:20:18.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:20:18.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:20:18.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:20:19.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T15:20:19.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:20:19.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:20:19.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-09-13T15:20:19.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-13T15:20:19.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-13T15:20:19.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-13T15:20:19.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-13T15:20:19.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-13T15:20:19.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-13T15:20:19.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-13T15:20:20.559 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'ecpool' created 2024-09-13T15:20:20.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:20:21.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-13T15:20:21.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-13T15:20:21.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:20:21.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:20:21.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:20:21.587 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:20:21.587 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:20:21.587 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:20:21.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:20:21.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:20:21.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:20:21.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:20:21.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:20:21.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:20:21.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:20:21.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:20:21.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:20:22.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:20:22.093 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:20:22.093 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:20:22.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:20:22.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:20:22.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:20:22.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-13T15:20:22.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-13T15:20:22.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-13T15:20:22.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:20:22.243 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:20:22.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672973 2024-09-13T15:20:22.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672973 2024-09-13T15:20:22.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672973' 2024-09-13T15:20:22.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:20:22.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:20:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-13T15:20:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-13T15:20:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672973 2-64424509448' 2024-09-13T15:20:22.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:20:22.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-13T15:20:22.530 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:20:22.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:20:22.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-13T15:20:22.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:20:22.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-13T15:20:22.533 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836502 2024-09-13T15:20:22.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-13T15:20:22.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:20:22.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-13T15:20:22.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:20:22.856 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672973 2024-09-13T15:20:22.856 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:20:22.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:20:22.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672973 2024-09-13T15:20:22.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:20:22.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672973 2024-09-13T15:20:22.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-13T15:20:22.861 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672973 2024-09-13T15:20:22.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:20:23.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-13T15:20:23.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:20:23.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-13T15:20:23.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:20:23.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:20:23.159 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-13T15:20:23.159 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:20:23.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-13T15:20:23.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-13T15:20:23.161 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509448 2024-09-13T15:20:23.162 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:20:23.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509448 2024-09-13T15:20:23.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:20:24.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:20:24.461 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:20:24.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-13T15:20:24.785 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:20:24.786 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:20:24.786 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:20:25.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T15:20:25.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:20:25.182 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:20:25.182 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:20:25.182 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:20:25.182 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:20:25.182 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:20:25.182 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:20:25.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T15:20:25.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:20:25.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:20:25.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:20:25.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T15:20:25.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:20:25.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:20:25.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-13T15:20:25.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-09-13T15:20:25.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:20:25.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:20:25.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-09-13T15:20:25.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:20:25.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:20:25.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:20:26.291 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:20:26.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:20:26.985 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:20:27.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:20:27.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:20:27.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:20:27.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-09-13T15:20:27.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-13T15:20:27.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ0 2024-09-13T15:20:27.259 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ0 2024-09-13T15:20:27.260 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T15:20:27.578 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-13T15:20:27.578 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:20:27.578 INFO:tasks.workunit.client.0.smithi112.stderr:0' 2024-09-13T15:20:27.578 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2024-09-13T15:20:27.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-09-13T15:20:27.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-09-13T15:20:27.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-09-13T15:20:27.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:20:27.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:20:27.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-09-13T15:20:27.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:20:27.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:20:27.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:20:28.014 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:20:28.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:20:28.516 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:20:28.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:20:28.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:20:28.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:20:28.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-09-13T15:20:28.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-13T15:20:28.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ1 2024-09-13T15:20:28.849 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ1 2024-09-13T15:20:28.849 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T15:20:29.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-13T15:20:29.184 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:20:29.184 INFO:tasks.workunit.client.0.smithi112.stderr:0' 2024-09-13T15:20:29.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2024-09-13T15:20:29.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-09-13T15:20:29.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-09-13T15:20:29.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-09-13T15:20:29.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:20:29.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:20:29.188 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:20:29.188 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:20:29.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:20:29.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:20:29.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:20:29.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:20:29.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:20:29.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:20:29.498 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:20:29.498 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:20:29.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:20:29.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:20:29.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:20:30.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:20:30.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:20:30.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-13T15:20:30.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:20:30.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:20:30.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:20:30.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:20:30.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:20:31.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:20:31.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-09-13T15:20:31.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-09-13T15:20:32.669 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-09-13T15:20:33.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-09-13T15:20:33.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-09-13T15:20:34.740 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-09-13T15:20:35.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-09-13T15:20:35.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-09-13T15:20:36.410 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-09-13T15:20:36.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-09-13T15:20:36.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-09-13T15:20:38.355 INFO:tasks.workunit.client.0.smithi112.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-09-13T15:20:38.893 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-09-13T15:20:38.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:20:38.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-09-13T15:20:38.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:20:38.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:20:38.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:20:38.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:20:38.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:20:38.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:20:38.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:20:38.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:20:38.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:20:38.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:20:38.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:20:38.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:20:38.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:20:38.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:20:38.899 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:20:38.899 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:20:38.899 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:20:38.899 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:20:38.899 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:20:38.899 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:20:38.899 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:20:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:20:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:20:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:20:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:20:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:20:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:20:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:20:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:20:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:20:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:20:38.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:20:38.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:20:38.901 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:20:38.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:20:38.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:20:38.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:20:38.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:20:38.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:20:38.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:20:38.935 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:20:38.930+0000 7f2d1b06a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:20:38.956 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:20:38.954+0000 7f2d1b06a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:20:38.957 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:20:38.954+0000 7f2d1b06a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:20:39.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:20:39.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:20:39.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:20:39.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:20:39.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:20:39.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:39.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:20:39.242 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:20:39.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:39.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:20:39.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:40.042 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:20:40.038+0000 7f2d1b06a7c0 -1 Falling back to public interface 2024-09-13T15:20:40.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:40.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:40.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:20:40.596 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:20:40.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:40.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:20:40.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:41.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:41.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:41.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:20:41.923 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:20:41.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:41.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:20:42.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:42.853 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:20:42.850+0000 7f2d1b06a7c0 -1 osd.0 35 log_to_monitors true 2024-09-13T15:20:43.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:43.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:43.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:20:43.252 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:20:43.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:43.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:20:43.664 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 42 up_thru 42 down_at 36 last_clean_interval [5,35) [v2:127.0.0.1:6802/118809917,v1:127.0.0.1:6803/118809917] [v2:127.0.0.1:6804/118809917,v1:127.0.0.1:6805/118809917] exists,up 2a8f688e-3e1d-4070-9780-dbb2b2a8ea34 2024-09-13T15:20:43.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:20:43.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:20:43.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:20:43.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:20:43.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-09-13T15:20:43.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:20:43.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:20:43.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:20:43.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:20:43.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:20:43.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:20:43.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:20:43.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:20:43.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:20:43.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:20:43.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:20:43.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:20:43.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:20:43.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:20:43.666 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:20:43.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:20:43.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:20:43.667 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:20:43.667 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:20:43.667 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:20:43.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:20:43.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:20:43.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:20:43.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:20:43.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:20:43.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:20:43.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:20:43.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:20:43.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:20:43.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:20:43.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:20:43.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:20:43.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:20:43.669 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:20:43.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:20:43.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:20:43.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:20:43.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:20:43.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:20:43.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:20:43.697 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:20:43.694+0000 7f80863707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:20:43.699 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:20:43.698+0000 7f80863707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:20:43.701 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:20:43.698+0000 7f80863707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:20:44.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:20:44.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:20:44.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:20:44.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:20:44.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:20:44.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:44.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:20:44.015 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:20:44.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:44.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:20:44.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:44.770 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:20:44.766+0000 7f80863707c0 -1 Falling back to public interface 2024-09-13T15:20:45.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:45.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:45.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:20:45.360 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:20:45.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:45.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:20:45.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:46.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:46.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:46.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:20:46.679 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:20:46.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:46.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:20:47.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:47.428 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:20:47.426+0000 7f80863707c0 -1 osd.1 37 log_to_monitors true 2024-09-13T15:20:48.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:48.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:48.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:20:48.003 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:20:48.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:48.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:20:48.416 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 38 last_clean_interval [10,37) [v2:127.0.0.1:6810/1116955236,v1:127.0.0.1:6811/1116955236] [v2:127.0.0.1:6812/1116955236,v1:127.0.0.1:6813/1116955236] exists,up b022a19b-2276-4df6-a43b-d9d2f107f669 2024-09-13T15:20:48.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:20:48.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:20:48.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:20:48.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:20:48.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-09-13T15:20:48.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:20:48.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:20:48.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T15:20:48.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:20:48.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:20:48.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:20:48.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:20:48.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:20:48.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:20:48.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T15:20:48.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T15:20:48.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:20:48.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:20:48.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:20:48.420 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:20:48.420 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:20:48.420 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:20:48.420 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:20:48.420 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:20:48.420 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:20:48.420 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:20:48.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:20:48.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:20:48.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:20:48.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:20:48.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:20:48.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:20:48.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:20:48.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:20:48.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:20:48.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:20:48.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T15:20:48.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T15:20:48.421 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.2 2024-09-13T15:20:48.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:20:48.422 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-13T15:20:48.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T15:20:48.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:20:48.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:20:48.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:20:48.453 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:20:48.450+0000 7fa45465f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:20:48.491 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:20:48.486+0000 7fa45465f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:20:48.492 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:20:48.490+0000 7fa45465f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:20:48.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T15:20:48.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:20:48.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:20:48.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:20:48.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:20:48.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:48.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:20:48.770 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:20:48.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:48.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:20:49.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:50.046 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:20:50.042+0000 7fa45465f7c0 -1 Falling back to public interface 2024-09-13T15:20:50.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:50.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:50.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:20:50.095 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:20:50.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:50.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:20:50.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:51.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:51.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:51.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:20:51.420 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:20:51.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:51.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:20:51.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:52.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:52.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:52.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:20:52.759 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:20:52.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:52.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:20:53.088 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:20:53.086+0000 7fa45465f7c0 -1 osd.2 39 log_to_monitors true 2024-09-13T15:20:53.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:20:54.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:20:54.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:20:54.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:20:54.112 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:20:54.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:20:54.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:20:54.517 INFO:tasks.workunit.client.0.smithi112.stderr:osd.2 up in weight 1 up_from 49 up_thru 49 down_at 40 last_clean_interval [15,39) [v2:127.0.0.1:6818/2712256042,v1:127.0.0.1:6819/2712256042] [v2:127.0.0.1:6820/2712256042,v1:127.0.0.1:6821/2712256042] exists,up 12a91fc7-df44-4dff-a272-14e32ffbd180 2024-09-13T15:20:54.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:20:54.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:20:54.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:20:54.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-09-13T15:20:54.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T15:20:55.013 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' removed 2024-09-13T15:20:55.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T15:20:55.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T15:20:55.541 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T15:20:55.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:20:56.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T15:20:58.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-09-13T15:20:58.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:20:58.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:20:58.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:20:58.223 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:20:58.224 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:20:58.224 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:20:58.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:20:58.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:20:58.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:20:58.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:20:58.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:20:58.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:20:58.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:20:58.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:20:58.361 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:20:58.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:20:58.736 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:20:58.737 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:20:58.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:20:58.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:20:58.737 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:20:58.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=180388626437 2024-09-13T15:20:58.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 180388626437 2024-09-13T15:20:58.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-180388626437' 2024-09-13T15:20:58.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:20:58.903 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:20:59.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528324 2024-09-13T15:20:59.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528324 2024-09-13T15:20:59.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-180388626437 1-193273528324' 2024-09-13T15:20:59.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:20:59.046 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:20:59.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=210453397507 2024-09-13T15:20:59.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 210453397507 2024-09-13T15:20:59.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-180388626437 1-193273528324 2-210453397507' 2024-09-13T15:20:59.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:20:59.193 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-180388626437 2024-09-13T15:20:59.193 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:20:59.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:20:59.195 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-180388626437 2024-09-13T15:20:59.195 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:20:59.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=180388626437 2024-09-13T15:20:59.196 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 180388626437 2024-09-13T15:20:59.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 180388626437' 2024-09-13T15:20:59.197 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:20:59.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 180388626436 -lt 180388626437 2024-09-13T15:20:59.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:21:00.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:21:00.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:21:00.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 180388626437 -lt 180388626437 2024-09-13T15:21:00.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:21:00.836 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-193273528324 2024-09-13T15:21:00.836 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:21:00.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:21:00.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:21:00.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-193273528324 2024-09-13T15:21:00.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528324 2024-09-13T15:21:00.840 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 193273528324 2024-09-13T15:21:00.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 193273528324' 2024-09-13T15:21:00.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:21:01.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528324 -lt 193273528324 2024-09-13T15:21:01.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:21:01.180 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-210453397507 2024-09-13T15:21:01.180 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:21:01.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:21:01.183 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:21:01.183 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-210453397507 2024-09-13T15:21:01.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=210453397507 2024-09-13T15:21:01.185 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 210453397507 2024-09-13T15:21:01.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 210453397507' 2024-09-13T15:21:01.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:21:01.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 210453397507 -lt 210453397507 2024-09-13T15:21:01.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:21:01.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:21:01.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:21:01.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T15:21:01.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:21:01.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:21:01.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:21:01.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:21:01.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:21:01.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:21:01.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:21:02.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T15:21:02.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:21:02.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:21:02.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:21:02.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T15:21:02.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:21:02.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:21:02.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-09-13T15:21:02.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T15:21:02.689 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=MOBJ0 2024-09-13T15:21:02.689 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool MOBJ0 2024-09-13T15:21:02.689 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T15:21:03.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-09-13T15:21:03.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-09-13T15:21:03.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T15:21:03.005 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T15:21:03.005 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T15:21:03.005 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:21:03.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:21:03.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T15:21:03.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T15:20:20.442366+0000 2024-09-13T15:21:03.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T15:21:03.606 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-09-13T15:21:03.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T15:20:20.442366+0000 2024-09-13T15:21:03.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T15:21:03.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:20:20.442366+0000 2024-09-13T15:21:03.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:21:03.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:21:03.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:21:03.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T15:21:03.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T15:21:03.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:21:03.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:21:03.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T15:21:03.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:20:20.442366+0000 '>' 2024-09-13T15:20:20.442366+0000 2024-09-13T15:21:03.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:21:04.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:21:04.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:21:04.922 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T15:21:04.922 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T15:21:04.922 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:21:04.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:21:04.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T15:21:05.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:20:20.442366+0000 '>' 2024-09-13T15:20:20.442366+0000 2024-09-13T15:21:05.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:21:06.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:21:06.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:21:06.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T15:21:06.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T15:21:06.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:21:06.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:21:06.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T15:21:06.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:20:20.442366+0000 '>' 2024-09-13T15:20:20.442366+0000 2024-09-13T15:21:06.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:21:07.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:21:07.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:21:07.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T15:21:07.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T15:21:07.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:21:07.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T15:21:07.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:21:07.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:20:20.442366+0000 '>' 2024-09-13T15:20:20.442366+0000 2024-09-13T15:21:07.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:21:08.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:21:08.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:21:08.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T15:21:08.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T15:21:08.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:21:08.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:21:08.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T15:21:09.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:21:04.094045+0000 '>' 2024-09-13T15:20:20.442366+0000 2024-09-13T15:21:09.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:21:09.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T15:21:11.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-09-13T15:21:11.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-09-13T15:21:11.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-09-13T15:21:11.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-09-13T15:21:11.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-09-13T15:21:11.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-09-13T15:21:11.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-09-13T15:21:11.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-09-13T15:21:11.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-09-13T15:21:11.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T15:21:11.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:21:11.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:21:11.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:21:11.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:21:11.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:21:11.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:21:11.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:21:11.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:21:11.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:21:11.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:21:11.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:21:11.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:21:11.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:21:11.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:21:11.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:21:11.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:21:11.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:21:11.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:21:11.712 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:21:11.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:21:11.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:21:11.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:21:11.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:21:11.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:21:11.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:21:11.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:21:11.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:21:11.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:21:11.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T15:21:11.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T15:21:11.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T15:21:11.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T15:21:11.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:21:11.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:21:11.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:21:11.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:21:11.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:21:11.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:21:11.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:21:11.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:21:11.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:21:11.746 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:21:11.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:21:11.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:21:11.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:21:11.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:21:11.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:21:11.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:21:11.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:21:11.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:21:11.752 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:21:11.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:21:11.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:21:11.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:21:11.757 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:21:11.757 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:21:11.757 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:21:11.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:21:11.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:21:11.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:21:11.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T15:21:11.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:21:11.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:21:11.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:21:11.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T15:21:11.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:21:11.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T15:21:11.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:21:11.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T15:21:11.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-09-13T15:21:11.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-09-13T15:21:11.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-09-13T15:21:11.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-09-13T15:21:11.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=true 2024-09-13T15:21:11.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-09-13T15:21:11.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-13T15:21:11.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T15:21:11.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:21:11.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:21:11.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:21:11.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T15:21:11.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T15:21:12.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:21:12.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:21:12.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:21:12.226 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:21:12.226 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:21:12.226 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:21:12.226 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:21:12.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T15:21:12.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:21:12.477 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:21:12.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:21:12.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:21:12.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:21:12.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:21:12.479 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:21:12.479 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:21:12.479 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:21:12.479 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:21:12.480 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:21:12.480 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:21:12.480 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:21:12.480 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:21:12.480 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T15:21:12.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:21:12.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:21:12.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:21:12.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:21:12.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:21:12.599 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:21:12.599 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:21:12.599 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:21:12.599 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:21:12.599 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:21:12.599 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:21:12.600 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:21:12.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:21:12.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T15:21:12.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-13T15:21:12.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T15:21:12.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:21:12.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:21:12.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:21:12.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T15:21:12.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:21:13.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:21:13.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:21:13.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:21:13.634 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:21:13.634 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:21:13.634 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:21:13.634 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:21:13.634 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:21:13.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:21:13.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-09-13T15:21:13.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:21:13.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-09-13T15:21:13.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:21:13.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:21:13.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:21:13.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:21:13.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:21:13.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:21:13.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:21:13.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:21:13.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:21:13.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:21:13.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:21:13.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:21:13.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:21:13.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:21:13.676 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:21:13.677 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:21:13.677 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:21:13.677 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:21:13.677 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:21:13.677 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:21:13.677 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:21:13.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:21:13.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:21:13.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:21:13.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:21:13.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:21:13.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:21:13.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:21:13.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:21:13.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:21:13.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:21:13.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:21:13.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:21:13.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:21:13.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:21:13.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=93819999-7ffe-49ac-869d-97d18a9f1f6e 2024-09-13T15:21:13.680 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 93819999-7ffe-49ac-869d-97d18a9f1f6e 2024-09-13T15:21:13.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 93819999-7ffe-49ac-869d-97d18a9f1f6e' 2024-09-13T15:21:13.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:21:13.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBpWORmArGYKRAAvj1kmBIALJ/nyTLVTlPMGQ== 2024-09-13T15:21:13.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBpWORmArGYKRAAvj1kmBIALJ/nyTLVTlPMGQ=="}' 2024-09-13T15:21:13.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 93819999-7ffe-49ac-869d-97d18a9f1f6e -i td/osd-scrub-repair/0/new.json 2024-09-13T15:21:13.989 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:21:14.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T15:21:14.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpWORmArGYKRAAvj1kmBIALJ/nyTLVTlPMGQ== --osd-uuid 93819999-7ffe-49ac-869d-97d18a9f1f6e 2024-09-13T15:21:14.045 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:21:14.041+0000 7f098286a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:21:14.073 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:21:14.069+0000 7f098286a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:21:14.074 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:21:14.069+0000 7f098286a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:21:14.074 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:21:14.069+0000 7f098286a7c0 -1 bdev(0x55b9bfb8fc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:21:14.075 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:21:14.069+0000 7f098286a7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T15:21:21.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T15:21:21.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:21:21.996 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:21:21.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:21:21.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:21:22.404 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:21:22.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:21:22.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:21:22.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:21:22.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:21:22.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:21:22.438 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:21:22.433+0000 7fd4c149a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:21:22.481 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:21:22.477+0000 7fd4c149a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:21:22.482 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:21:22.477+0000 7fd4c149a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:21:22.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:21:22.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:21:22.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:21:22.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:21:22.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:21:22.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:22.727 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:21:22.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:21:22.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:22.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:23.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:23.542 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:21:23.537+0000 7fd4c149a7c0 -1 Falling back to public interface 2024-09-13T15:21:24.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:24.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:24.053 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:21:24.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:21:24.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:24.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:24.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:25.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:25.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:25.372 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:21:25.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:21:25.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:25.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:25.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:26.628 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:21:26.625+0000 7fd4c149a7c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:21:26.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:26.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:26.695 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:21:26.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:21:26.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:26.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:27.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:28.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:28.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:28.249 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:21:28.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:21:28.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:28.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:28.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:29.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:29.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:29.714 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:21:29.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:21:29.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:29.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:30.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:31.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:31.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:31.044 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:21:31.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:21:31.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:31.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:31.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:32.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:32.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:32.401 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:21:32.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:21:32.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:32.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:32.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:33.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:33.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:33.870 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:21:33.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:21:33.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:33.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:34.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:35.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:35.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:35.200 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:21:35.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:21:35.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:35.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:35.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:36.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:36.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:36.721 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:21:36.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:21:36.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:36.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:37.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:38.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:38.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:38.065 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:21:38.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:21:38.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:38.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:38.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:39.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:39.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:39.401 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:21:39.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:21:39.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:39.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:39.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:40.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:40.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:40.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:21:40.840 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:21:40.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:40.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:41.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:42.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:42.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:42.279 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:21:42.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:21:42.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:42.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:42.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:43.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:43.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:43.777 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:21:43.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:21:43.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:43.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:44.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:45.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:45.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:45.116 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:21:45.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:21:45.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:45.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:45.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:46.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:46.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:46.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:21:46.702 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:21:46.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:46.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:47.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:48.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:48.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:48.754 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:21:48.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:21:48.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:48.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:49.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:50.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:50.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:50.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:21:50.678 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:21:50.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:50.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:51.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:52.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:52.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:52.399 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:21:52.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:21:52.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:52.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:53.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:54.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:54.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:54.293 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:21:54.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:21:54.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:54.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:54.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:55.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:55.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:55.781 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:21:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:21:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:56.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:57.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:57.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:57.335 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:21:57.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:21:57.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:57.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:57.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:21:58.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:21:58.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:21:58.969 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:21:58.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:21:58.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:21:58.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:21:59.354 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1070949225,v1:127.0.0.1:6803/1070949225] [v2:127.0.0.1:6804/1070949225,v1:127.0.0.1:6805/1070949225] exists,up 93819999-7ffe-49ac-869d-97d18a9f1f6e 2024-09-13T15:21:59.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:21:59.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:21:59.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:21:59.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:21:59.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-09-13T15:21:59.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:21:59.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:21:59.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T15:21:59.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:21:59.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:21:59.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:21:59.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:21:59.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:21:59.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:21:59.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:21:59.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:21:59.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:21:59.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:21:59.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:21:59.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:21:59.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:21:59.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:21:59.359 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:21:59.359 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:21:59.359 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:21:59.359 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:21:59.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:21:59.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:21:59.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:21:59.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:21:59.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:21:59.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:21:59.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:21:59.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:21:59.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:21:59.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:21:59.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:21:59.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:21:59.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:21:59.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:21:59.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=63e683e6-e2bc-46b0-9a2e-5fc69b71a141 2024-09-13T15:21:59.361 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 63e683e6-e2bc-46b0-9a2e-5fc69b71a141 2024-09-13T15:21:59.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 63e683e6-e2bc-46b0-9a2e-5fc69b71a141' 2024-09-13T15:21:59.361 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:21:59.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCXWORma+HWFhAAP/zXcXBqFKV1X4elyghUrQ== 2024-09-13T15:21:59.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCXWORma+HWFhAAP/zXcXBqFKV1X4elyghUrQ=="}' 2024-09-13T15:21:59.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 63e683e6-e2bc-46b0-9a2e-5fc69b71a141 -i td/osd-scrub-repair/1/new.json 2024-09-13T15:21:59.804 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:21:59.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T15:21:59.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCXWORma+HWFhAAP/zXcXBqFKV1X4elyghUrQ== --osd-uuid 63e683e6-e2bc-46b0-9a2e-5fc69b71a141 2024-09-13T15:21:59.860 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:21:59.860+0000 7fcfcd17c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:21:59.948 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:21:59.948+0000 7fcfcd17c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:21:59.950 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:21:59.948+0000 7fcfcd17c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:21:59.950 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:21:59.948+0000 7fcfcd17c7c0 -1 bdev(0x55fd046ffc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T15:21:59.950 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:21:59.948+0000 7fcfcd17c7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T15:22:08.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T15:22:08.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:22:08.306 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T15:22:08.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T15:22:08.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:22:08.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T15:22:08.705 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:22:08.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:22:08.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:22:08.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:22:08.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:22:08.738 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:22:08.736+0000 7f62623e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:22:08.778 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:22:08.776+0000 7f62623e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:22:08.779 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:22:08.780+0000 7f62623e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:22:09.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T15:22:09.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:22:09.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:22:09.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:22:09.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:22:09.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:09.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:22:09.024 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:22:09.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:09.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:09.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:10.314 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:22:10.312+0000 7f62623e27c0 -1 Falling back to public interface 2024-09-13T15:22:10.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:10.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:10.353 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:22:10.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:22:10.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:10.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:10.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:11.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:11.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:11.690 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:22:11.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:22:11.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:11.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:12.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:13.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:13.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:13.013 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:22:13.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:22:13.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:13.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:13.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:13.484 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:22:13.484+0000 7f62623e27c0 -1 osd.1 0 log_to_monitors true 2024-09-13T15:22:14.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:14.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:14.330 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:22:14.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:22:14.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:14.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:14.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:15.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:15.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:15.780 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:22:15.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:22:15.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:15.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:16.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:17.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:17.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:17.329 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:22:17.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:22:17.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:17.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:17.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:18.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:18.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:18.764 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:22:18.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:22:18.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:18.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:19.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:20.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:20.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:20.166 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:22:20.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:22:20.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:20.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:20.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:21.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:21.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:21.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:22:21.500 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:22:21.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:21.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:22.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:23.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:23.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:23.085 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:22:23.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:22:23.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:23.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:23.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:24.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:24.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:24.537 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:22:24.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:22:24.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:24.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:25.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:26.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:26.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:26.130 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:22:26.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:22:26.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:26.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:26.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:27.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:27.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:27.467 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:22:27.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:22:27.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:27.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:27.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:28.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:28.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:28.940 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:22:28.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:22:28.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:28.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:29.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:30.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:30.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:30.462 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:22:30.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:22:30.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:30.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:31.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:32.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:32.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:32.069 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:22:32.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:22:32.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:32.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:32.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:33.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:33.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:33.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:22:33.468 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:22:33.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:33.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:33.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:34.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:34.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:34.801 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:22:34.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:22:34.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:34.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:35.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:36.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:36.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:36.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:22:36.541 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:22:36.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:36.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:37.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:38.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:38.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:38.310 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:22:38.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:22:38.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:38.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:38.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:39.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:39.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:39.821 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:22:39.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:22:39.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:39.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:40.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:41.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:41.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:41.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:22:41.354 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:22:41.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:41.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:42.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:43.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:43.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:43.013 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:22:43.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:22:43.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:43.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:43.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:44.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:44.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:44.717 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:22:44.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:22:44.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:44.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:45.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:46.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:46.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:46.737 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:22:46.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:22:46.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:46.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:22:47.094 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1108065946,v1:127.0.0.1:6811/1108065946] [v2:127.0.0.1:6812/1108065946,v1:127.0.0.1:6813/1108065946] exists,up 63e683e6-e2bc-46b0-9a2e-5fc69b71a141 2024-09-13T15:22:47.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:22:47.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:22:47.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:22:47.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:22:47.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-09-13T15:22:47.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:22:47.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:22:47.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T15:22:47.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:22:47.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:22:47.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:22:47.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:22:47.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:22:47.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:22:47.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T15:22:47.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T15:22:47.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:22:47.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:22:47.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:22:47.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:22:47.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:22:47.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:22:47.097 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:22:47.099 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:22:47.099 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:22:47.099 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:22:47.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:22:47.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:22:47.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:22:47.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:22:47.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:22:47.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:22:47.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:22:47.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:22:47.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:22:47.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:22:47.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:22:47.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:22:47.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T15:22:47.100 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:22:47.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c5230cdd-38e1-4208-88ce-6733a27f72c6 2024-09-13T15:22:47.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c5230cdd-38e1-4208-88ce-6733a27f72c6' 2024-09-13T15:22:47.101 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 c5230cdd-38e1-4208-88ce-6733a27f72c6 2024-09-13T15:22:47.101 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:22:47.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDHWORmnwxaBxAABkV6g82J/ca4cji2LEh4zA== 2024-09-13T15:22:47.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDHWORmnwxaBxAABkV6g82J/ca4cji2LEh4zA=="}' 2024-09-13T15:22:47.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c5230cdd-38e1-4208-88ce-6733a27f72c6 -i td/osd-scrub-repair/2/new.json 2024-09-13T15:22:47.546 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:22:47.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T15:22:47.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDHWORmnwxaBxAABkV6g82J/ca4cji2LEh4zA== --osd-uuid c5230cdd-38e1-4208-88ce-6733a27f72c6 2024-09-13T15:22:47.602 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:22:47.599+0000 7fbc68ac97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:22:47.680 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:22:47.679+0000 7fbc68ac97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:22:47.682 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:22:47.679+0000 7fbc68ac97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:22:47.683 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:22:47.679+0000 7fbc68ac97c0 -1 bdev(0x559d88937c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T15:22:47.683 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:22:47.679+0000 7fbc68ac97c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T15:22:55.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T15:22:55.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:22:55.996 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T15:22:55.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T15:22:55.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:22:56.415 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T15:22:56.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T15:22:56.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:22:56.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:22:56.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:22:56.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:22:56.448 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:22:56.447+0000 7f0bf52207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:22:56.504 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:22:56.503+0000 7f0bf52207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:22:56.506 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:22:56.503+0000 7f0bf52207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:22:56.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T15:22:56.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:22:56.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:22:56.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:22:56.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:22:56.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:56.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:22:56.721 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:22:56.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:56.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:22:57.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:57.566 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:22:57.562+0000 7f0bf52207c0 -1 Falling back to public interface 2024-09-13T15:22:58.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:58.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:58.043 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:22:58.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:22:58.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:58.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:22:58.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:22:59.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:22:59.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:22:59.362 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:22:59.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:22:59.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:22:59.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:22:59.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:00.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:00.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:00.696 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:23:00.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:23:00.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:00.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:00.702 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:23:00.698+0000 7f0bf52207c0 -1 osd.2 0 log_to_monitors true 2024-09-13T15:23:01.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:02.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:02.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:02.033 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:23:02.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:23:02.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:02.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:02.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:03.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:03.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:03.358 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:23:03.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:23:03.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:03.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:03.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:04.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:04.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:04.839 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:23:04.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:23:04.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:04.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:06.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:06.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:06.214 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:23:06.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:23:06.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:06.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:06.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:07.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:07.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:07.589 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:23:07.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:23:07.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:07.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:08.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:09.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:09.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:09.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:23:09.084 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:23:09.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:09.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:09.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:10.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:10.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:10.535 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:23:10.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:23:10.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:10.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:11.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:12.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:12.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:12.201 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:23:12.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:23:12.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:12.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:12.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:13.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:13.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:13.597 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:23:13.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:23:13.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:13.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:14.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:15.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:15.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:15.105 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:23:15.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:23:15.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:15.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:15.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:16.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:16.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:16.544 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:23:16.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:23:16.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:16.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:17.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:18.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:18.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:18.155 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:23:18.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:23:18.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:18.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:18.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:19.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:19.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:19.606 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:23:19.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:23:19.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:19.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:20.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:21.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:21.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:21.456 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:23:21.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:23:21.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:21.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:22.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:23.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:23.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:23.049 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:23:23.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:23:23.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:23.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:23.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:24.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:24.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:24.802 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:23:24.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:23:24.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:24.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:25.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:26.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:26.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:26.121 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:23:26.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:23:26.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:26.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:26.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:27.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:27.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:27.605 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:23:27.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:23:27.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:27.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:28.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:29.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:29.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:29.252 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:23:29.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:23:29.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:29.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:29.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:31.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:31.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:31.001 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:23:31.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:23:31.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:31.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:31.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:32.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:32.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:32.681 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:23:32.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:23:32.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:32.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:33.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:23:34.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:23:34.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:23:34.087 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:23:34.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:23:34.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:23:34.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:23:34.424 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2551474096,v1:127.0.0.1:6819/2551474096] [v2:127.0.0.1:6820/2551474096,v1:127.0.0.1:6821/2551474096] exists,up c5230cdd-38e1-4208-88ce-6733a27f72c6 2024-09-13T15:23:34.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:23:34.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:23:34.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:23:34.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-09-13T15:23:34.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T15:23:34.892 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' does not exist 2024-09-13T15:23:34.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T15:23:34.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T15:23:35.377 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T15:23:35.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:23:36.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T15:23:39.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-09-13T15:23:39.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:23:39.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:23:39.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:23:39.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:23:39.669 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:23:39.669 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:23:39.669 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:23:39.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:23:39.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:23:39.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:23:39.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:23:39.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:23:39.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:23:39.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:23:39.809 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:23:40.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:23:40.152 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:23:40.152 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:23:40.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:23:40.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:23:40.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:23:40.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-13T15:23:40.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-13T15:23:40.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-13T15:23:40.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:23:40.343 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:23:40.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705680 2024-09-13T15:23:40.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705680 2024-09-13T15:23:40.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-38654705680' 2024-09-13T15:23:40.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:23:40.487 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:23:40.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574860 2024-09-13T15:23:40.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574860 2024-09-13T15:23:40.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-38654705680 2-55834574860' 2024-09-13T15:23:40.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:23:40.634 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-13T15:23:40.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:23:40.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:23:40.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-13T15:23:40.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:23:40.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-13T15:23:40.638 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836504 2024-09-13T15:23:40.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-13T15:23:40.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:23:41.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-13T15:23:41.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:23:41.008 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705680 2024-09-13T15:23:41.008 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:23:41.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:23:41.011 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705680 2024-09-13T15:23:41.011 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:23:41.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705680 2024-09-13T15:23:41.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705680' 2024-09-13T15:23:41.013 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 38654705680 2024-09-13T15:23:41.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:23:41.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705679 -lt 38654705680 2024-09-13T15:23:41.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:23:42.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:23:42.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:23:42.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705681 -lt 38654705680 2024-09-13T15:23:42.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:23:42.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-55834574860 2024-09-13T15:23:42.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:23:42.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:23:42.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-55834574860 2024-09-13T15:23:42.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:23:42.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574860 2024-09-13T15:23:42.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 55834574860' 2024-09-13T15:23:42.745 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 55834574860 2024-09-13T15:23:42.745 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:23:43.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574860 -lt 55834574860 2024-09-13T15:23:43.082 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:23:43.082 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:23:43.082 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:23:43.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T15:23:43.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:23:43.466 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:23:43.466 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:23:43.466 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:23:43.466 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:23:43.467 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:23:43.467 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:23:43.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T15:23:43.867 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:23:43.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:23:43.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:23:44.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T15:23:44.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:23:44.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:23:44.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-09-13T15:23:44.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-13T15:23:44.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-13T15:23:44.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-13T15:23:44.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-13T15:23:44.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-13T15:23:45.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-13T15:23:45.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-13T15:23:46.180 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'ecpool' created 2024-09-13T15:23:46.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:23:47.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-13T15:23:47.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-13T15:23:48.096 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 2 allow_ec_overwrites to true 2024-09-13T15:23:48.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-13T15:23:48.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:23:48.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:23:48.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:23:48.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:23:48.124 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:23:48.124 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:23:48.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:23:48.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:23:48.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:23:48.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:23:48.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:23:48.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:23:48.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:23:48.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:23:48.258 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:23:48.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:23:48.712 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:23:48.712 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:23:48.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:23:48.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:23:48.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:23:48.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-13T15:23:48.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-13T15:23:48.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-13T15:23:48.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:23:48.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:23:49.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=38654705683 2024-09-13T15:23:49.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 38654705683 2024-09-13T15:23:49.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-38654705683' 2024-09-13T15:23:49.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:23:49.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:23:49.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574862 2024-09-13T15:23:49.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574862 2024-09-13T15:23:49.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-38654705683 2-55834574862' 2024-09-13T15:23:49.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:23:49.193 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:23:49.193 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-13T15:23:49.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:23:49.195 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-13T15:23:49.195 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:23:49.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-13T15:23:49.196 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836506 2024-09-13T15:23:49.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-13T15:23:49.197 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:23:49.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836506 2024-09-13T15:23:49.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:23:50.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:23:50.512 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:23:50.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836506 2024-09-13T15:23:50.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:23:50.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-38654705683 2024-09-13T15:23:50.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:23:50.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:23:50.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-38654705683 2024-09-13T15:23:50.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:23:50.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=38654705683 2024-09-13T15:23:50.921 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 38654705683 2024-09-13T15:23:50.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 38654705683' 2024-09-13T15:23:50.922 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:23:51.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 38654705683 -lt 38654705683 2024-09-13T15:23:51.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:23:51.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-55834574862 2024-09-13T15:23:51.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:23:51.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:23:51.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-55834574862 2024-09-13T15:23:51.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:23:51.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574862 2024-09-13T15:23:51.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 55834574862' 2024-09-13T15:23:51.261 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 55834574862 2024-09-13T15:23:51.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:23:51.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574863 -lt 55834574862 2024-09-13T15:23:51.636 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:23:51.636 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:23:51.636 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:23:52.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T15:23:52.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:23:52.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:23:52.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:23:52.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:23:52.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:23:52.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:23:52.093 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:23:52.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T15:23:52.464 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:23:52.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:23:52.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:23:52.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T15:23:52.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:23:52.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:23:52.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-13T15:23:52.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-09-13T15:23:52.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:23:52.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:23:52.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-09-13T15:23:52.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:23:52.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:23:52.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:23:53.336 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:23:53.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:23:53.880 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:23:53.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:23:53.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:23:53.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:23:54.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-09-13T15:23:54.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-13T15:23:54.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ0 2024-09-13T15:23:54.222 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ0 2024-09-13T15:23:54.222 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T15:23:54.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-13T15:23:54.661 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:23:54.661 INFO:tasks.workunit.client.0.smithi112.stderr:0' 2024-09-13T15:23:54.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2024-09-13T15:23:54.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-09-13T15:23:54.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-09-13T15:23:54.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-09-13T15:23:54.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:23:54.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:23:54.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-09-13T15:23:54.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:23:54.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:23:54.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:23:55.285 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:23:55.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:23:55.803 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:23:55.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:23:55.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:23:55.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:23:56.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-09-13T15:23:56.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-13T15:23:56.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ1 2024-09-13T15:23:56.212 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T15:23:56.213 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ1 2024-09-13T15:23:56.548 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2024-09-13T15:23:56.548 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:23:56.548 INFO:tasks.workunit.client.0.smithi112.stderr:0' 2024-09-13T15:23:56.548 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2024-09-13T15:23:56.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-09-13T15:23:56.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-09-13T15:23:56.548 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-09-13T15:23:56.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:23:56.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:23:56.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:23:56.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:23:56.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:23:56.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:23:56.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:23:57.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:23:57.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:23:57.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:23:57.866 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:23:57.866 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:23:57.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:23:57.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:23:57.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:23:59.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:23:59.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:23:59.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-13T15:23:59.183 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:23:59.183 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:23:59.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:23:59.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:23:59.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:24:00.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:24:00.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-09-13T15:24:00.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-09-13T15:24:02.451 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-09-13T15:24:02.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-09-13T15:24:02.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-09-13T15:24:05.239 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-09-13T15:24:05.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-09-13T15:24:05.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-09-13T15:24:07.368 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-09-13T15:24:07.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-09-13T15:24:07.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-09-13T15:24:09.972 INFO:tasks.workunit.client.0.smithi112.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-09-13T15:24:10.506 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-09-13T15:24:10.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:24:10.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-09-13T15:24:10.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:24:10.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:24:10.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:24:10.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:24:10.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:24:10.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:24:10.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:24:10.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:24:10.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:24:10.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:24:10.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:24:10.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:24:10.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:24:10.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:24:10.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:24:10.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:24:10.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:24:10.511 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:24:10.511 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:24:10.511 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:24:10.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:24:10.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:24:10.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:24:10.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:24:10.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:24:10.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:24:10.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:24:10.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:24:10.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:24:10.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:24:10.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:24:10.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:24:10.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:24:10.514 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:24:10.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:24:10.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:24:10.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:24:10.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:24:10.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:24:10.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:24:10.553 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:10.549+0000 7f62670a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:24:10.556 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:10.553+0000 7f62670a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:24:10.558 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:10.553+0000 7f62670a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:24:10.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:24:10.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:24:10.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:24:10.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:24:10.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:24:10.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:24:10.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:24:10.848 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:24:10.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:24:10.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:24:11.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:24:11.842 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:11.837+0000 7f62670a87c0 -1 Falling back to public interface 2024-09-13T15:24:12.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:24:12.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:24:12.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:24:12.218 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:24:12.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:24:12.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:24:12.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:24:13.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:24:13.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:24:13.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:24:13.556 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:24:13.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:24:13.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:24:13.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:24:14.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:24:14.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:24:14.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:24:14.873 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:24:14.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:24:14.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:24:15.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:24:15.356 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:15.353+0000 7f62670a87c0 -1 osd.0 33 log_to_monitors true 2024-09-13T15:24:16.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:24:16.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:24:16.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:24:16.211 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:24:16.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:24:16.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:24:16.579 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 41 up_thru 41 down_at 34 last_clean_interval [5,33) [v2:127.0.0.1:6802/3547066735,v1:127.0.0.1:6803/3547066735] [v2:127.0.0.1:6804/3547066735,v1:127.0.0.1:6805/3547066735] exists,up 93819999-7ffe-49ac-869d-97d18a9f1f6e 2024-09-13T15:24:16.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:24:16.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:24:16.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:24:16.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:24:16.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-09-13T15:24:16.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:24:16.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:24:16.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:24:16.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:24:16.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:24:16.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:24:16.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:24:16.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:24:16.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:24:16.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:24:16.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:24:16.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:24:16.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:24:16.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:24:16.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:24:16.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:24:16.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:24:16.581 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:24:16.581 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:24:16.581 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:24:16.583 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:24:16.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:24:16.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:24:16.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:24:16.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:24:16.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:24:16.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:24:16.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:24:16.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:24:16.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:24:16.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:24:16.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:24:16.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:24:16.584 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T15:24:16.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:24:16.584 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:24:16.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:24:16.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:24:16.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:24:16.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:24:16.610 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:16.605+0000 7fd2dde4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:24:16.656 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:16.653+0000 7fd2dde4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:24:16.657 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:16.653+0000 7fd2dde4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:24:16.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:24:16.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:24:16.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:24:16.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:24:16.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:24:16.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:24:16.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:24:16.913 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:24:16.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:24:16.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:24:17.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:24:17.942 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:17.937+0000 7fd2dde4b7c0 -1 Falling back to public interface 2024-09-13T15:24:18.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:24:18.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:24:18.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:24:18.240 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:24:18.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:24:18.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:24:18.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:24:19.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:24:19.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:24:19.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:24:19.578 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:24:19.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:24:19.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:24:19.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:24:20.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:24:20.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:24:20.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:24:20.916 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:24:20.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:24:20.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:24:20.999 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:20.996+0000 7fd2dde4b7c0 -1 osd.1 36 log_to_monitors true 2024-09-13T15:24:21.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:24:22.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:24:22.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:24:22.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:24:22.290 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:24:22.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:24:22.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:24:22.695 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 37 last_clean_interval [9,36) [v2:127.0.0.1:6810/2199960776,v1:127.0.0.1:6811/2199960776] [v2:127.0.0.1:6812/2199960776,v1:127.0.0.1:6813/2199960776] exists,up 63e683e6-e2bc-46b0-9a2e-5fc69b71a141 2024-09-13T15:24:22.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:24:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:24:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:24:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-09-13T15:24:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-09-13T15:24:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:24:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:24:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T15:24:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:24:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:24:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:24:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:24:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:24:22.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:24:22.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T15:24:22.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T15:24:22.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:24:22.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:24:22.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:24:22.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:24:22.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:24:22.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:24:22.699 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:24:22.699 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:24:22.699 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:24:22.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:24:22.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:24:22.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:24:22.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:24:22.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:24:22.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:24:22.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:24:22.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:24:22.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:24:22.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:24:22.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:24:22.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T15:24:22.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T15:24:22.701 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.2 2024-09-13T15:24:22.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:24:22.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-13T15:24:22.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T15:24:22.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:24:22.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:24:22.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:24:22.732 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:22.728+0000 7f905f2807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:24:22.768 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:22.764+0000 7f905f2807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:24:22.770 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:22.764+0000 7f905f2807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:24:23.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T15:24:23.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:24:23.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:24:23.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:24:23.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:24:23.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:24:23.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:24:23.140 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:24:23.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:24:23.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:24:23.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:24:23.814 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:23.808+0000 7f905f2807c0 -1 Falling back to public interface 2024-09-13T15:24:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:24:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:24:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:24:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:24:24.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:24:24.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:24:24.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:24:25.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:24:25.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:24:25.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:24:25.823 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:24:25.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:24:25.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:24:26.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:24:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:24:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:24:27.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:24:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:24:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:24:27.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:24:27.198 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:27.192+0000 7f905f2807c0 -1 osd.2 38 log_to_monitors true 2024-09-13T15:24:27.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:24:28.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:24:28.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:24:28.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:24:28.615 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:24:28.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:24:28.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:24:29.033 INFO:tasks.workunit.client.0.smithi112.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 39 last_clean_interval [13,38) [v2:127.0.0.1:6818/2546236088,v1:127.0.0.1:6819/2546236088] [v2:127.0.0.1:6820/2546236088,v1:127.0.0.1:6821/2546236088] exists,up c5230cdd-38e1-4208-88ce-6733a27f72c6 2024-09-13T15:24:29.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:24:29.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:24:29.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:24:29.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-09-13T15:24:29.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T15:24:29.759 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' removed 2024-09-13T15:24:29.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T15:24:29.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T15:24:30.529 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T15:24:30.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:24:31.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T15:24:34.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-09-13T15:24:34.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:24:34.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:24:34.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:24:34.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:24:34.385 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:24:34.385 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:24:34.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:24:34.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:24:34.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:24:34.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:24:34.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:24:34.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:24:34.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:24:34.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:24:34.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:24:34.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:24:34.854 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:24:34.854 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:24:34.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:24:34.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:24:34.855 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:24:35.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659141 2024-09-13T15:24:35.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659141 2024-09-13T15:24:35.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-176093659141' 2024-09-13T15:24:35.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:24:35.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:24:35.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=188978561028 2024-09-13T15:24:35.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 188978561028 2024-09-13T15:24:35.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-176093659141 1-188978561028' 2024-09-13T15:24:35.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:24:35.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:24:35.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462915 2024-09-13T15:24:35.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462915 2024-09-13T15:24:35.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-176093659141 1-188978561028 2-201863462915' 2024-09-13T15:24:35.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:24:35.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-176093659141 2024-09-13T15:24:35.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:24:35.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:24:35.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-176093659141 2024-09-13T15:24:35.325 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:24:35.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659141 2024-09-13T15:24:35.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 176093659141' 2024-09-13T15:24:35.326 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 176093659141 2024-09-13T15:24:35.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:24:35.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659140 -lt 176093659141 2024-09-13T15:24:35.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:24:36.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:24:36.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:24:37.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659142 -lt 176093659141 2024-09-13T15:24:37.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:24:37.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-188978561028 2024-09-13T15:24:37.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:24:37.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:24:37.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:24:37.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-188978561028 2024-09-13T15:24:37.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=188978561028 2024-09-13T15:24:37.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 188978561028' 2024-09-13T15:24:37.025 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 188978561028 2024-09-13T15:24:37.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:24:37.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 188978561029 -lt 188978561028 2024-09-13T15:24:37.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:24:37.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-201863462915 2024-09-13T15:24:37.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:24:37.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:24:37.334 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-201863462915 2024-09-13T15:24:37.334 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:24:37.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462915 2024-09-13T15:24:37.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-09-13T15:24:37.335 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 201863462915 2024-09-13T15:24:37.336 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:24:37.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462915 -lt 201863462915 2024-09-13T15:24:37.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:24:37.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:24:37.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:24:38.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T15:24:38.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:24:38.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:24:38.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:24:38.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:24:38.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:24:38.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:24:38.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:24:38.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T15:24:38.314 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:24:38.314 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:24:38.315 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:24:38.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T15:24:38.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:24:38.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:24:38.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-09-13T15:24:38.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T15:24:38.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=MOBJ0 2024-09-13T15:24:38.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool MOBJ0 2024-09-13T15:24:38.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T15:24:39.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-09-13T15:24:39.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-09-13T15:24:39.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-13T15:24:39.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-13T15:24:39.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T15:24:39.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:24:39.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:24:39.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T15:24:39.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T15:23:45.871465+0000 2024-09-13T15:24:39.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-13T15:24:39.688 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-09-13T15:24:39.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-13T15:23:45.871465+0000 2024-09-13T15:24:39.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-13T15:24:39.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:23:45.871465+0000 2024-09-13T15:24:39.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:24:39.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:24:39.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:24:39.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T15:24:39.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T15:24:39.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:24:39.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:24:39.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T15:24:40.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:23:45.871465+0000 '>' 2024-09-13T15:23:45.871465+0000 2024-09-13T15:24:40.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:24:41.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:24:41.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:24:41.043 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T15:24:41.044 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T15:24:41.044 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:24:41.044 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T15:24:41.044 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:24:41.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:23:45.871465+0000 '>' 2024-09-13T15:23:45.871465+0000 2024-09-13T15:24:41.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:24:42.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:24:42.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:24:42.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T15:24:42.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T15:24:42.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:24:42.370 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:24:42.370 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T15:24:42.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:23:45.871465+0000 '>' 2024-09-13T15:23:45.871465+0000 2024-09-13T15:24:42.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:24:43.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:24:43.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:24:43.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T15:24:43.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T15:24:43.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:24:43.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:24:43.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T15:24:44.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:23:45.871465+0000 '>' 2024-09-13T15:23:45.871465+0000 2024-09-13T15:24:44.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:24:45.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:24:45.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:24:45.082 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-13T15:24:45.082 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-13T15:24:45.082 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:24:45.082 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:24:45.083 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-13T15:24:45.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:24:40.179072+0000 '>' 2024-09-13T15:23:45.871465+0000 2024-09-13T15:24:45.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:24:45.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T15:24:47.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-09-13T15:24:47.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-09-13T15:24:47.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-09-13T15:24:47.420 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-09-13T15:24:47.420 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-09-13T15:24:47.420 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-09-13T15:24:47.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-09-13T15:24:47.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-09-13T15:24:47.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-09-13T15:24:47.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T15:24:47.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:24:47.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:24:47.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:24:47.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:24:47.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:24:47.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:24:47.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:24:47.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:24:47.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:24:47.888 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:24:47.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:24:47.891 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:24:47.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:24:47.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:24:47.893 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:24:47.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:24:47.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:24:47.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:24:47.895 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:24:47.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:24:47.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:24:47.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:24:47.922 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:24:47.922 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:24:47.922 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:24:47.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:24:47.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:24:47.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:24:47.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T15:24:47.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T15:24:47.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T15:24:47.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T15:24:47.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:24:47.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:24:47.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:24:47.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:24:47.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:24:47.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:24:47.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:24:47.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:24:47.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:24:47.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:24:47.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:24:47.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:24:47.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:24:47.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:24:47.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:24:47.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:24:47.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:24:47.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:24:47.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:24:47.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:24:47.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:24:47.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:24:47.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:24:47.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:24:47.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:24:47.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:24:47.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:24:47.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:24:47.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T15:24:47.941 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:24:47.942 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:24:47.942 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:24:47.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T15:24:47.943 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:24:47.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T15:24:47.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:24:47.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T15:24:47.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-09-13T15:24:47.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5760: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-09-13T15:24:47.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5761: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-09-13T15:24:47.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5762: TEST_periodic_scrub_replicated: local objname=POBJ 2024-09-13T15:24:47.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_periodic_scrub_replicated: return 0 2024-09-13T15:24:47.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T15:24:47.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:24:47.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:24:47.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:24:47.946 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:24:47.946 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:24:47.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:24:47.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:24:47.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:24:47.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:24:47.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:24:47.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:24:47.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:24:47.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:24:47.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:24:47.952 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:24:47.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:24:47.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:24:47.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:24:47.955 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:24:47.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:24:47.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:24:47.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:24:47.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:24:47.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:24:47.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:24:47.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:24:47.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:24:47.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:24:47.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T15:24:47.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T15:24:47.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T15:24:47.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T15:24:47.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:24:47.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:24:47.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:24:47.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:24:47.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:24:47.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:24:47.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:24:47.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:24:47.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:24:47.965 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:24:47.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:24:47.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:24:47.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:24:47.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:24:47.969 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:24:47.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:24:47.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:24:47.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:24:47.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:24:47.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:24:47.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:24:47.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:24:47.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:24:47.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:24:47.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:24:47.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:24:47.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:24:47.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:24:47.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T15:24:47.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:24:47.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:24:47.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:24:47.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T15:24:47.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:24:47.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T15:24:47.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:24:47.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T15:24:47.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2024-09-13T15:24:47.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-09-13T15:24:47.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local poolname=testpool 2024-09-13T15:24:47.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local OSDS=2 2024-09-13T15:24:47.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OBJS=30 2024-09-13T15:24:47.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: local REPAIRS=20 2024-09-13T15:24:47.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-09-13T15:24:47.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T15:24:47.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:24:47.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:24:47.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:24:47.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T15:24:47.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T15:24:48.504 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:24:48.504 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:24:48.504 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:24:48.505 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:24:48.505 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:24:48.505 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:24:48.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:24:48.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T15:24:48.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:24:48.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:24:48.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:24:48.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:24:48.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:24:48.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:24:48.744 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:24:48.744 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:24:48.744 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:24:48.745 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:24:48.745 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:24:48.745 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:24:48.745 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:24:48.745 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:24:48.746 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T15:24:48.867 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:24:48.867 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:24:48.867 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:24:48.867 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:24:48.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:24:48.868 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:24:48.868 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:24:48.868 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:24:48.869 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:24:48.869 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:24:48.869 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:24:48.869 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:24:48.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:24:48.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T15:24:48.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-09-13T15:24:48.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T15:24:48.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:24:48.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:24:48.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:24:48.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T15:24:48.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:24:50.666 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:24:50.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:24:50.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:24:50.667 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:24:50.667 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:24:50.667 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:24:50.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:24:50.669 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:24:50.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:24:50.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:665: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T15:24:50.706 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: expr 2 - 1 2024-09-13T15:24:50.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: seq 0 1 2024-09-13T15:24:50.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:24:50.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:24:50.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:24:50.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:24:50.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:24:50.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:24:50.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:24:50.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:24:50.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:24:50.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:24:50.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:24:50.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:24:50.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:24:50.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:24:50.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:24:50.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:24:50.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:24:50.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:24:50.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:24:50.713 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:24:50.713 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:24:50.713 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:24:50.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:24:50.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:24:50.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:24:50.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:24:50.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:24:50.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:24:50.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:24:50.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:24:50.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:24:50.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:24:50.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:24:50.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:24:50.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T15:24:50.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:24:50.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:24:50.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7874edc7-a3ec-48d8-a8f0-81cbb6dc46c6 2024-09-13T15:24:50.715 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 7874edc7-a3ec-48d8-a8f0-81cbb6dc46c6 2024-09-13T15:24:50.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7874edc7-a3ec-48d8-a8f0-81cbb6dc46c6' 2024-09-13T15:24:50.716 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:24:50.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBCWeRmej+oKxAAekneNcYDq2iK7keWCjp50Q== 2024-09-13T15:24:50.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBCWeRmej+oKxAAekneNcYDq2iK7keWCjp50Q=="}' 2024-09-13T15:24:50.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7874edc7-a3ec-48d8-a8f0-81cbb6dc46c6 -i td/osd-scrub-repair/0/new.json 2024-09-13T15:24:51.035 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:24:51.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T15:24:51.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBCWeRmej+oKxAAekneNcYDq2iK7keWCjp50Q== --osd-uuid 7874edc7-a3ec-48d8-a8f0-81cbb6dc46c6 2024-09-13T15:24:51.091 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:51.092+0000 7f5db098e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:24:51.119 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:51.120+0000 7f5db098e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:24:51.121 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:51.120+0000 7f5db098e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:24:51.121 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:51.120+0000 7f5db098e7c0 -1 bdev(0x55faa2d39c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:24:51.121 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:51.120+0000 7f5db098e7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T15:24:59.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T15:24:59.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:24:59.237 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:24:59.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:24:59.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:24:59.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:24:59.636 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:24:59.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:24:59.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:24:59.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:24:59.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:24:59.669 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:59.668+0000 7f3f196c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:24:59.707 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:59.708+0000 7f3f196c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:24:59.709 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:24:59.708+0000 7f3f196c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:25:00.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:25:00.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:25:00.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:25:00.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:25:00.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:25:00.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:00.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:25:00.019 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:25:00.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:00.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:00.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:00.998 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:25:00.995+0000 7f3f196c47c0 -1 Falling back to public interface 2024-09-13T15:25:01.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:01.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:01.344 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:25:01.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:25:01.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:01.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:01.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:02.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:02.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:02.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:25:02.680 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:25:02.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:02.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:02.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:04.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:04.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:04.001 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:25:04.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:25:04.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:04.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:04.008 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:25:04.007+0000 7f3f196c47c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:25:04.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:05.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:05.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:05.400 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:25:05.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:25:05.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:05.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:05.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:06.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:06.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:06.727 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:25:06.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:25:06.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:06.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:07.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:08.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:08.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:08.139 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:25:08.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:25:08.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:08.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:08.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:09.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:09.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:09.472 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:25:09.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:25:09.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:09.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:09.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:10.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:10.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:10.837 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:25:10.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:25:10.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:10.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:11.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:12.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:12.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:12.363 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:25:12.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:25:12.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:12.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:12.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:13.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:13.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:13.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:25:13.702 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:25:13.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:13.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:14.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:15.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:15.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:15.123 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:25:15.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:25:15.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:15.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:15.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:16.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:16.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:16.440 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:25:16.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:25:16.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:16.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:16.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:17.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:17.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:17.867 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:25:17.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:25:17.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:17.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:18.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:19.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:19.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:19.295 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:25:19.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:25:19.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:19.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:19.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:20.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:20.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:20.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:25:20.655 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:25:20.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:20.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:21.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:22.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:22.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:22.118 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:25:22.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:25:22.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:22.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:22.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:23.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:23.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:23.443 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:25:23.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:25:23.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:23.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:24.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:25.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:25.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:25.623 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:25:25.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:25:25.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:25.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:26.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:27.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:27.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:27.417 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:25:27.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:25:27.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:27.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:28.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:29.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:29.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:29.136 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:25:29.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:25:29.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:29.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:30.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:31.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:31.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:31.148 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:25:31.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:25:31.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:31.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:31.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:32.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:32.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:32.764 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:25:32.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:25:32.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:32.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:33.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:34.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:34.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:34.576 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:25:34.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:25:34.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:34.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:34.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:35.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:35.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:35.898 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:25:35.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:25:35.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:35.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:25:36.223 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/629236765,v1:127.0.0.1:6803/629236765] [v2:127.0.0.1:6804/629236765,v1:127.0.0.1:6805/629236765] exists,up 7874edc7-a3ec-48d8-a8f0-81cbb6dc46c6 2024-09-13T15:25:36.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:25:36.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:25:36.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:25:36.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:25:36.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:25:36.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:25:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:25:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T15:25:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:25:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:25:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:25:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:25:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:25:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:25:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:25:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:25:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:25:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:25:36.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:25:36.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:25:36.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:25:36.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:25:36.229 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:25:36.229 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:25:36.229 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:25:36.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:25:36.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:25:36.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:25:36.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:25:36.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:25:36.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:25:36.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:25:36.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:25:36.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:25:36.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:25:36.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:25:36.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:25:36.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T15:25:36.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:25:36.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:25:36.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=96c241f1-73e5-4d21-a443-6a9bbe6a34f4 2024-09-13T15:25:36.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 96c241f1-73e5-4d21-a443-6a9bbe6a34f4' 2024-09-13T15:25:36.231 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 96c241f1-73e5-4d21-a443-6a9bbe6a34f4 2024-09-13T15:25:36.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:25:36.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBwWeRmmwO/DhAAOmG9Zs2tTrgSUwvMJs4X5A== 2024-09-13T15:25:36.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBwWeRmmwO/DhAAOmG9Zs2tTrgSUwvMJs4X5A=="}' 2024-09-13T15:25:36.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 96c241f1-73e5-4d21-a443-6a9bbe6a34f4 -i td/osd-scrub-repair/1/new.json 2024-09-13T15:25:36.725 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:25:36.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T15:25:36.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBwWeRmmwO/DhAAOmG9Zs2tTrgSUwvMJs4X5A== --osd-uuid 96c241f1-73e5-4d21-a443-6a9bbe6a34f4 2024-09-13T15:25:36.774 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:25:36.771+0000 7f2d2206a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:25:36.869 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:25:36.867+0000 7f2d2206a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:25:36.871 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:25:36.871+0000 7f2d2206a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:25:36.872 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:25:36.871+0000 7f2d2206a7c0 -1 bdev(0x561ee6577c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T15:25:36.872 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:25:36.871+0000 7f2d2206a7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T15:25:45.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T15:25:45.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:25:45.378 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T15:25:45.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T15:25:45.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:25:45.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T15:25:45.775 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:25:45.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:25:45.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:25:45.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:25:45.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:25:45.809 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:25:45.806+0000 7f36a3f857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:25:45.822 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:25:45.818+0000 7f36a3f857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:25:45.823 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:25:45.822+0000 7f36a3f857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:25:46.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T15:25:46.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:25:46.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:25:46.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:25:46.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:25:46.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:46.113 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:25:46.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:25:46.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:46.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:25:46.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:47.081 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:25:47.078+0000 7f36a3f857c0 -1 Falling back to public interface 2024-09-13T15:25:47.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:47.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:47.425 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:25:47.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:25:47.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:47.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:25:47.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:48.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:48.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:48.752 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:25:48.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:25:48.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:48.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:25:49.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:50.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:50.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:50.096 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:25:50.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:25:50.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:50.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:25:50.278 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:25:50.274+0000 7f36a3f857c0 -1 osd.1 0 log_to_monitors true 2024-09-13T15:25:50.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:51.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:51.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:51.429 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:25:51.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:25:51.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:51.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:25:52.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:53.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:53.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:53.035 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:25:53.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:25:53.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:53.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:25:53.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:54.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:54.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:54.488 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:25:54.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:25:54.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:54.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:25:54.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:55.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:55.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:55.889 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:25:55.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:25:55.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:55.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:25:56.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:57.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:57.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:57.595 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:25:57.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:25:57.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:57.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:25:57.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:25:58.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:25:58.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:25:58.928 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:25:58.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:25:58.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:25:58.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:25:59.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:00.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:00.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:00.261 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:26:00.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:26:00.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:00.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:00.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:01.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:01.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:01.660 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:26:01.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:26:01.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:01.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:02.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:03.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:03.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:03.063 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:26:03.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:26:03.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:03.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:03.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:04.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:04.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:04.474 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:26:04.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:26:04.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:04.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:04.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:05.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:05.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:05.792 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:26:05.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:26:05.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:05.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:06.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:07.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:07.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:07.123 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:26:07.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:26:07.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:07.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:07.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:08.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:08.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:08.465 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:26:08.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:26:08.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:08.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:08.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:09.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:09.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:09.894 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:26:09.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:26:09.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:09.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:10.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:11.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:11.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:11.260 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:26:11.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:26:11.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:11.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:11.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:12.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:12.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:12.735 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:26:12.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:26:12.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:12.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:13.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:14.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:14.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:14.285 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:26:14.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:26:14.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:14.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:15.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:16.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:16.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:16.047 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:26:16.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:26:16.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:16.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:16.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:17.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:17.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:17.848 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:26:17.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:26:17.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:17.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:18.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:19.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:19.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:19.551 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:26:19.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:26:19.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:19.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:20.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:21.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:21.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:21.109 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:26:21.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:26:21.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:21.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:21.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:22.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:22.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:22.784 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:26:22.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:26:22.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:22.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:23.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:26:24.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:26:24.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:26:24.122 INFO:tasks.workunit.client.0.smithi112.stdout:26 2024-09-13T15:26:24.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 26 2024-09-13T15:26:24.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:26:24.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:26:24.454 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3041611696,v1:127.0.0.1:6811/3041611696] [v2:127.0.0.1:6812/3041611696,v1:127.0.0.1:6813/3041611696] exists,up 96c241f1-73e5-4d21-a443-6a9bbe6a34f4 2024-09-13T15:26:24.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:26:24.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:26:24.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:26:24.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:670: TEST_repair_stats: create_pool testpool 1 1 2024-09-13T15:26:24.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-13T15:26:24.946 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'testpool' created 2024-09-13T15:26:24.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:26:25.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: ceph osd pool set testpool size 2 2024-09-13T15:26:26.887 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 1 size to 2 2024-09-13T15:26:26.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: wait_for_clean 2024-09-13T15:26:26.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:26:26.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:26:26.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:26:26.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:26:26.911 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:26:26.911 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:26:26.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:26:26.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:26:26.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:26:27.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:26:27.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:26:27.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:26:27.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:26:27.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:26:27.043 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:26:27.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:26:27.499 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:26:27.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:26:27.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:26:27.499 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:26:27.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836497 2024-09-13T15:26:27.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836497 2024-09-13T15:26:27.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836497' 2024-09-13T15:26:27.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:26:27.714 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:26:27.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-13T15:26:27.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-13T15:26:27.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836497 1-42949672965' 2024-09-13T15:26:27.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:26:27.878 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836497 2024-09-13T15:26:27.878 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:26:27.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:26:27.880 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836497 2024-09-13T15:26:27.880 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:26:27.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836497 2024-09-13T15:26:27.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-09-13T15:26:27.882 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836497 2024-09-13T15:26:27.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:26:28.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836496 -lt 21474836497 2024-09-13T15:26:28.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:26:29.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:26:29.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:26:29.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836497 2024-09-13T15:26:29.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:26:29.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-13T15:26:29.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:26:29.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:26:29.712 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-13T15:26:29.712 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:26:29.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-13T15:26:29.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-13T15:26:29.714 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672965 2024-09-13T15:26:29.714 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:26:30.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672966 -lt 42949672965 2024-09-13T15:26:30.044 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:26:30.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:26:30.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:26:30.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:26:30.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:26:30.445 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:26:30.445 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:26:30.445 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:26:30.445 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:26:30.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:26:30.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:26:30.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:26:30.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:26:30.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:26:30.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:26:31.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:26:31.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:26:31.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:26:31.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:675: TEST_repair_stats: local payload=ABCDEF 2024-09-13T15:26:31.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: echo ABCDEF 2024-09-13T15:26:31.155 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: seq 1 30 2024-09-13T15:26:31.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:31.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:31.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:31.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:31.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:31.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:31.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:31.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:31.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:31.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:32.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:32.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:32.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:32.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:32.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:32.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:32.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:32.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:32.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:32.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:33.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:33.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:33.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:33.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:33.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:33.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:33.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:33.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:33.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:33.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:34.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:34.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:34.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:34.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:34.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:34.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:34.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:34.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:34.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:34.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:35.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:35.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:35.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:35.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:35.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:35.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:35.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:35.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:35.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:35.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:36.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:36.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:36.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:36.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:36.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:36.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:36.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:36.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:37.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-09-13T15:26:37.154 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:26:37.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: get_not_primary testpool obj1 2024-09-13T15:26:37.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-13T15:26:37.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-13T15:26:37.348 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2024-09-13T15:26:37.349 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T15:26:37.349 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T15:26:37.349 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-13T15:26:37.349 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:26:37.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T15:26:37.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2024-09-13T15:26:37.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T15:26:38.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: local other=0 2024-09-13T15:26:38.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_pg testpool obj1 2024-09-13T15:26:38.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-13T15:26:38.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2024-09-13T15:26:38.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T15:26:38.016 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2024-09-13T15:26:38.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local pgid=1.0 2024-09-13T15:26:38.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_primary testpool obj1 2024-09-13T15:26:38.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T15:26:38.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T15:26:38.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-13T15:26:38.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:26:38.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local primary=1 2024-09-13T15:26:38.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:688: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:26:38.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:26:38.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:26:38.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:26:38.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:26:38.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:26:40.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:26:40.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:26:40.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:26:40.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:26:40.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:26:40.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:26:40.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:26:41.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:26:41.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: seq 1 20 2024-09-13T15:26:41.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:26:41.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 1 % 2 2024-09-13T15:26:41.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-13T15:26:41.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-09-13T15:26:41.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:26:41.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:26:41.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:26:41.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:26:41.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:26:41.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-09-13T15:26:43.384 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-13T15:26:43.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:26:43.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 2 % 2 2024-09-13T15:26:43.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-13T15:26:43.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-13T15:26:43.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:26:43.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:26:43.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:26:43.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:26:43.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:26:43.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-13T15:26:46.129 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:104778fc:::obj2:head# 2024-09-13T15:26:46.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:26:46.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 3 % 2 2024-09-13T15:26:46.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-13T15:26:46.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-09-13T15:26:46.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:26:46.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:26:46.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:26:46.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:26:46.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:26:46.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-09-13T15:26:48.293 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:8dd16f86:::obj3:head# 2024-09-13T15:26:48.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:26:48.829 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 4 % 2 2024-09-13T15:26:48.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-13T15:26:48.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-09-13T15:26:48.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:26:48.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:26:48.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:26:48.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:26:48.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:26:48.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-09-13T15:26:50.405 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:0ee9ae15:::obj4:head# 2024-09-13T15:26:50.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:26:50.942 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 5 % 2 2024-09-13T15:26:50.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-13T15:26:50.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-09-13T15:26:50.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:26:50.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:26:50.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:26:50.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:26:50.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:26:50.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-09-13T15:26:52.424 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:c52c9666:::obj5:head# 2024-09-13T15:26:52.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:26:52.960 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 6 % 2 2024-09-13T15:26:52.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-13T15:26:52.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-09-13T15:26:52.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:26:52.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:26:52.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:26:52.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:26:52.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:26:52.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-09-13T15:26:54.401 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:233a42c1:::obj6:head# 2024-09-13T15:26:54.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:26:54.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 7 % 2 2024-09-13T15:26:54.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-13T15:26:54.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-09-13T15:26:54.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:26:54.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:26:54.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:26:54.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:26:54.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:26:54.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-09-13T15:26:56.311 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:9a09113a:::obj7:head# 2024-09-13T15:26:56.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:26:56.858 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 8 % 2 2024-09-13T15:26:56.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-13T15:26:56.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-09-13T15:26:56.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:26:56.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:26:56.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:26:56.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:26:56.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:26:56.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-09-13T15:26:58.272 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:a557efb1:::obj8:head# 2024-09-13T15:26:58.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:26:58.816 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 9 % 2 2024-09-13T15:26:58.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-13T15:26:58.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-09-13T15:26:58.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:26:58.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:26:58.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:26:58.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:26:58.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:26:58.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-09-13T15:27:00.208 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:fe351e27:::obj9:head# 2024-09-13T15:27:00.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:27:00.754 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 10 % 2 2024-09-13T15:27:00.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-13T15:27:00.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-09-13T15:27:00.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:27:00.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:27:00.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:27:00.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:27:00.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:27:00.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-09-13T15:27:02.294 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:d1337354:::obj10:head# 2024-09-13T15:27:02.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:27:02.828 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 11 % 2 2024-09-13T15:27:02.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-13T15:27:02.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-09-13T15:27:02.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:27:02.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:27:02.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:27:02.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:27:02.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:27:02.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-09-13T15:27:04.229 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:eeae2a94:::obj11:head# 2024-09-13T15:27:05.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:27:05.286 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 12 % 2 2024-09-13T15:27:05.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-13T15:27:05.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-09-13T15:27:05.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:27:05.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:27:05.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:27:05.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:27:05.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:27:05.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-09-13T15:27:06.568 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:af5f95cb:::obj12:head# 2024-09-13T15:27:07.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:27:07.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 13 % 2 2024-09-13T15:27:07.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-13T15:27:07.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-09-13T15:27:07.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:27:07.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:27:07.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:27:07.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:27:07.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:27:07.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-09-13T15:27:08.763 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:a61ad63e:::obj13:head# 2024-09-13T15:27:09.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:27:09.306 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 14 % 2 2024-09-13T15:27:09.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-13T15:27:09.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-09-13T15:27:09.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:27:09.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:27:09.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:27:09.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:27:09.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:27:09.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-09-13T15:27:10.723 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:e0a44829:::obj14:head# 2024-09-13T15:27:11.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:27:11.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 15 % 2 2024-09-13T15:27:11.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-13T15:27:11.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-09-13T15:27:11.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:27:11.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:27:11.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:27:11.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:27:11.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:27:11.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-09-13T15:27:12.425 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:ab946124:::obj15:head# 2024-09-13T15:27:12.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:27:12.969 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 16 % 2 2024-09-13T15:27:12.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-13T15:27:12.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-09-13T15:27:12.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:27:12.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:27:12.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:27:12.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:27:12.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:27:12.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-09-13T15:27:14.227 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:461f8b5e:::obj16:head# 2024-09-13T15:27:14.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:27:14.766 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 17 % 2 2024-09-13T15:27:14.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-13T15:27:14.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-09-13T15:27:14.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:27:14.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:27:14.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:27:14.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:27:14.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:27:14.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-09-13T15:27:16.113 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:24fc6e92:::obj17:head# 2024-09-13T15:27:16.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:27:16.649 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 18 % 2 2024-09-13T15:27:16.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-13T15:27:16.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-09-13T15:27:16.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:27:16.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:27:16.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:27:16.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:27:16.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:27:16.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-09-13T15:27:17.938 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:b9836a99:::obj18:head# 2024-09-13T15:27:18.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:27:18.480 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 19 % 2 2024-09-13T15:27:18.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-09-13T15:27:18.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-09-13T15:27:18.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:27:18.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:27:18.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:27:18.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:27:18.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:27:18.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-09-13T15:27:19.730 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:00b31b6c:::obj19:head# 2024-09-13T15:27:20.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-09-13T15:27:20.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 20 % 2 2024-09-13T15:27:20.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-09-13T15:27:20.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-09-13T15:27:20.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:27:20.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:27:20.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:27:20.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:27:20.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:27:20.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-09-13T15:27:21.427 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:24b5611b:::obj20:head# 2024-09-13T15:27:21.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:27:21.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:27:21.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:27:21.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:27:21.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:27:21.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:27:21.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:27:21.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:27:21.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:27:21.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:27:21.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:27:21.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:27:21.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:27:21.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:27:21.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:27:21.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:27:21.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:27:21.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:27:21.968 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:27:21.968 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:27:21.968 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:27:21.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:27:21.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:27:21.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:27:21.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:27:21.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:27:21.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:27:21.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:27:21.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:27:21.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:27:21.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:27:21.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T15:27:21.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:27:21.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:27:21.971 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:27:21.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:27:21.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:27:21.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:27:21.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:27:21.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:27:21.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:27:22.008 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:27:22.008+0000 7f5ba66b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:27:22.010 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:27:22.012+0000 7f5ba66b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:27:22.012 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:27:22.012+0000 7f5ba66b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:27:22.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:27:22.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:27:22.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:27:22.308 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:27:22.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:27:22.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:27:22.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:27:22.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:27:22.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:27:22.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:27:22.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:27:23.273 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:27:23.272+0000 7f5ba66b27c0 -1 Falling back to public interface 2024-09-13T15:27:23.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:27:23.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:27:23.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:27:23.685 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:27:23.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:27:23.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:27:24.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:27:25.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:27:25.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:27:25.044 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:27:25.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:27:25.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:27:25.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:27:25.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:27:26.310 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:27:26.308+0000 7f5ba66b27c0 -1 osd.1 18 log_to_monitors true 2024-09-13T15:27:26.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:27:26.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:27:26.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:27:26.381 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:27:26.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:27:26.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:27:26.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:27:27.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:27:27.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:27:27.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:27:27.753 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:27:27.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:27:27.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:27:28.121 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [10,18) [v2:127.0.0.1:6802/3205809583,v1:127.0.0.1:6803/3205809583] [v2:127.0.0.1:6804/3205809583,v1:127.0.0.1:6805/3205809583] exists,up 96c241f1-73e5-4d21-a443-6a9bbe6a34f4 2024-09-13T15:27:28.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:27:28.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:27:28.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:27:28.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:27:28.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:27:28.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:27:28.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:27:28.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:27:28.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:27:28.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:27:28.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:27:28.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:27:28.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:27:28.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:27:28.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:27:28.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:27:28.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:27:28.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:27:28.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:27:28.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:27:28.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:27:28.127 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:27:28.127 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:27:28.127 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:27:28.127 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:27:28.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:27:28.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:27:28.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:27:28.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:27:28.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:27:28.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:27:28.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:27:28.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:27:28.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:27:28.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T15:27:28.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:27:28.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:27:28.128 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:27:28.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:27:28.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:27:28.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:27:28.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:27:28.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:27:28.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:27:28.160 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:27:28.160+0000 7f7be9f057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:27:28.186 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:27:28.184+0000 7f7be9f057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:27:28.188 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:27:28.188+0000 7f7be9f057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:27:28.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:27:28.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:27:28.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:27:28.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:27:28.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:27:28.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:27:28.460 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:27:28.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:27:28.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:27:28.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:27:28.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:27:29.221 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:27:29.220+0000 7f7be9f057c0 -1 Falling back to public interface 2024-09-13T15:27:29.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:27:29.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:27:29.771 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:27:29.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:27:29.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:27:29.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:27:30.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:27:31.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:27:31.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:27:31.112 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:27:31.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:27:31.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:27:31.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:27:31.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:27:32.262 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:27:32.264+0000 7f7be9f057c0 -1 osd.0 16 log_to_monitors true 2024-09-13T15:27:32.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:27:32.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:27:32.456 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:27:32.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:27:32.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:27:32.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:27:32.838 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6810/251858648,v1:127.0.0.1:6811/251858648] [v2:127.0.0.1:6812/251858648,v1:127.0.0.1:6813/251858648] exists,up 7874edc7-a3ec-48d8-a8f0-81cbb6dc46c6 2024-09-13T15:27:32.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:27:32.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:27:32.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:27:32.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2024-09-13T15:27:32.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:27:32.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:27:32.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:27:32.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:27:32.840 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:27:32.840 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:27:32.840 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:27:32.840 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:27:32.840 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:27:32.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:27:32.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:27:32.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:27:32.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:27:32.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:27:32.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:27:33.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:27:33.283 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:27:33.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:27:33.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:27:33.284 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:27:33.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215106 2024-09-13T15:27:33.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215106 2024-09-13T15:27:33.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106' 2024-09-13T15:27:33.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:27:33.419 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:27:33.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=90194313219 2024-09-13T15:27:33.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 90194313219 2024-09-13T15:27:33.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106 1-90194313219' 2024-09-13T15:27:33.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:27:33.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-103079215106 2024-09-13T15:27:33.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:27:33.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:27:33.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-103079215106 2024-09-13T15:27:33.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:27:33.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215106 2024-09-13T15:27:33.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-09-13T15:27:33.580 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 103079215106 2024-09-13T15:27:33.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:27:34.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215106 -lt 103079215106 2024-09-13T15:27:34.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:27:34.104 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-90194313219 2024-09-13T15:27:34.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:27:34.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:27:34.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-90194313219 2024-09-13T15:27:34.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:27:34.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=90194313219 2024-09-13T15:27:34.110 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 90194313219 2024-09-13T15:27:34.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 90194313219' 2024-09-13T15:27:34.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:27:34.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 90194313218 -lt 90194313219 2024-09-13T15:27:34.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:27:35.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:27:35.486 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:27:35.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 90194313219 -lt 90194313219 2024-09-13T15:27:35.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:27:35.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:27:35.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:27:36.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:27:36.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:27:36.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:27:36.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:27:36.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:27:36.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:27:36.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:27:36.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:27:36.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T15:27:36.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:27:36.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:27:36.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:27:36.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T15:27:36.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T15:27:36.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:27:36.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T15:27:36.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:27:36.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:27:37.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:27:37.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:27:37.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:27:37.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:27:37.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:27:37.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:27:37.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:27:37.080 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:27:37.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T15:27:37.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:27:37.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:27:37.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:27:37.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T15:27:37.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T15:27:37.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:27:37.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:27:37.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:27:37.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:27:37.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:27:37.758 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:27:37.758 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:27:38.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:27:38.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:27:38.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T15:27:38.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:27:38.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T15:27:38.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:27:38.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:27:38.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:27:38.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:27:38.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:27:38.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:27:38.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:27:38.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:27:38.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:27:38.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:27:38.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:27:38.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:27:39.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:27:39.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:27:39.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:27:39.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: repair 1.0 2024-09-13T15:27:39.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-13T15:27:39.030 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-13T15:27:39.030 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:27:39.030 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:27:39.030 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:27:39.030 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:27:39.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T15:26:24.857246+0000 2024-09-13T15:27:39.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-13T15:27:39.643 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.0 on osd.1 to repair 2024-09-13T15:27:39.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-13T15:26:24.857246+0000 2024-09-13T15:27:39.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T15:27:39.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:26:24.857246+0000 2024-09-13T15:27:39.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:27:39.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:27:39.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:27:39.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:27:39.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:27:39.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:27:39.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:27:39.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:27:39.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:26:24.857246+0000 '>' 2024-09-13T15:26:24.857246+0000 2024-09-13T15:27:39.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:27:40.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:27:40.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:27:40.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:27:40.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:27:40.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:27:40.960 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:27:40.960 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:27:41.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:26:24.857246+0000 '>' 2024-09-13T15:26:24.857246+0000 2024-09-13T15:27:41.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:27:42.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:27:42.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:27:42.293 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:27:42.293 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:27:42.293 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:27:42.294 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:27:42.294 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:27:42.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:26:24.857246+0000 '>' 2024-09-13T15:26:24.857246+0000 2024-09-13T15:27:42.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:27:43.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:27:43.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:27:43.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:27:43.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:27:43.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:27:43.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:27:43.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:27:43.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:27:40.291344+0000 '>' 2024-09-13T15:26:24.857246+0000 2024-09-13T15:27:43.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:27:43.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T15:27:45.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: wait_for_clean 2024-09-13T15:27:45.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:27:45.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:27:45.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:27:45.965 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:27:45.966 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:27:45.966 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:27:45.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:27:45.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:27:45.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:27:46.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:27:46.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:27:46.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:27:46.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:27:46.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:27:46.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:27:46.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:27:46.435 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:27:46.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:27:46.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:27:46.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:27:46.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215109 2024-09-13T15:27:46.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215109 2024-09-13T15:27:46.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215109' 2024-09-13T15:27:46.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:27:46.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:27:46.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=90194313223 2024-09-13T15:27:46.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 90194313223 2024-09-13T15:27:46.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215109 1-90194313223' 2024-09-13T15:27:46.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:27:46.753 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-103079215109 2024-09-13T15:27:46.753 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:27:46.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:27:46.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-103079215109 2024-09-13T15:27:46.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:27:46.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215109 2024-09-13T15:27:46.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 103079215109' 2024-09-13T15:27:46.756 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 103079215109 2024-09-13T15:27:46.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:27:47.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215108 -lt 103079215109 2024-09-13T15:27:47.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:27:48.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:27:48.081 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:27:48.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215109 -lt 103079215109 2024-09-13T15:27:48.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:27:48.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-90194313223 2024-09-13T15:27:48.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:27:48.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:27:48.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-90194313223 2024-09-13T15:27:48.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:27:48.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=90194313223 2024-09-13T15:27:48.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 90194313223' 2024-09-13T15:27:48.405 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 90194313223 2024-09-13T15:27:48.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:27:48.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 90194313223 -lt 90194313223 2024-09-13T15:27:48.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:27:48.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:27:48.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:27:49.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:27:49.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:27:49.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:27:49.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:27:49.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:27:49.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:27:49.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:27:49.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:27:49.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:27:49.497 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:27:49.497 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:27:49.497 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:27:49.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:27:49.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:27:49.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:27:49.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: ceph pg dump pgs 2024-09-13T15:27:50.222 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T15:27:50.222 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-09-13T15:27:41.911871+0000 16'30 25:129 [1,0] 1 [1,0] 1 16'30 2024-09-13T15:27:40.291344+0000 16'30 2024-09-13T15:27:40.291344+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:27:40.291344+0000 20 0 2024-09-13T15:27:50.222 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T15:27:50.222 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T15:27:50.222 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:27:50.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: flush_pg_stats 2024-09-13T15:27:50.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:27:50.240 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:27:50.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:27:50.565 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:27:50.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:27:50.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:27:50.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:27:50.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215111 2024-09-13T15:27:50.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215111 2024-09-13T15:27:50.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215111' 2024-09-13T15:27:50.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:27:50.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:27:50.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=90194313224 2024-09-13T15:27:50.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 90194313224 2024-09-13T15:27:50.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215111 1-90194313224' 2024-09-13T15:27:50.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:27:50.940 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:27:50.940 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-103079215111 2024-09-13T15:27:50.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:27:50.943 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-103079215111 2024-09-13T15:27:50.943 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:27:50.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215111 2024-09-13T15:27:50.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 103079215111' 2024-09-13T15:27:50.945 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 103079215111 2024-09-13T15:27:50.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:27:51.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215110 -lt 103079215111 2024-09-13T15:27:51.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:27:52.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:27:52.251 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:27:52.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215111 -lt 103079215111 2024-09-13T15:27:52.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:27:52.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-90194313224 2024-09-13T15:27:52.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:27:52.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:27:52.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-90194313224 2024-09-13T15:27:52.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:27:52.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=90194313224 2024-09-13T15:27:52.578 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 90194313224 2024-09-13T15:27:52.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 90194313224' 2024-09-13T15:27:52.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:27:52.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 90194313224 -lt 90194313224 2024-09-13T15:27:52.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: ceph pg 1.0 query 2024-09-13T15:27:52.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: jq .info.stats.stat_sum 2024-09-13T15:27:53.080 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:27:53.080 INFO:tasks.workunit.client.0.smithi112.stdout: "num_bytes": 210, 2024-09-13T15:27:53.080 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects": 30, 2024-09-13T15:27:53.080 INFO:tasks.workunit.client.0.smithi112.stdout: "num_object_clones": 0, 2024-09-13T15:27:53.080 INFO:tasks.workunit.client.0.smithi112.stdout: "num_object_copies": 60, 2024-09-13T15:27:53.080 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T15:27:53.081 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_missing": 0, 2024-09-13T15:27:53.081 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_degraded": 0, 2024-09-13T15:27:53.081 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_misplaced": 0, 2024-09-13T15:27:53.081 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_unfound": 0, 2024-09-13T15:27:53.081 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_dirty": 30, 2024-09-13T15:27:53.081 INFO:tasks.workunit.client.0.smithi112.stdout: "num_whiteouts": 0, 2024-09-13T15:27:53.081 INFO:tasks.workunit.client.0.smithi112.stdout: "num_read": 0, 2024-09-13T15:27:53.081 INFO:tasks.workunit.client.0.smithi112.stdout: "num_read_kb": 0, 2024-09-13T15:27:53.081 INFO:tasks.workunit.client.0.smithi112.stdout: "num_write": 30, 2024-09-13T15:27:53.081 INFO:tasks.workunit.client.0.smithi112.stdout: "num_write_kb": 30, 2024-09-13T15:27:53.081 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrub_errors": 0, 2024-09-13T15:27:53.082 INFO:tasks.workunit.client.0.smithi112.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T15:27:53.082 INFO:tasks.workunit.client.0.smithi112.stdout: "num_deep_scrub_errors": 0, 2024-09-13T15:27:53.082 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_recovered": 20, 2024-09-13T15:27:53.082 INFO:tasks.workunit.client.0.smithi112.stdout: "num_bytes_recovered": 140, 2024-09-13T15:27:53.082 INFO:tasks.workunit.client.0.smithi112.stdout: "num_keys_recovered": 0, 2024-09-13T15:27:53.082 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_omap": 0, 2024-09-13T15:27:53.082 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T15:27:53.082 INFO:tasks.workunit.client.0.smithi112.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T15:27:53.082 INFO:tasks.workunit.client.0.smithi112.stdout: "num_flush": 0, 2024-09-13T15:27:53.082 INFO:tasks.workunit.client.0.smithi112.stdout: "num_flush_kb": 0, 2024-09-13T15:27:53.082 INFO:tasks.workunit.client.0.smithi112.stdout: "num_evict": 0, 2024-09-13T15:27:53.082 INFO:tasks.workunit.client.0.smithi112.stdout: "num_evict_kb": 0, 2024-09-13T15:27:53.082 INFO:tasks.workunit.client.0.smithi112.stdout: "num_promote": 0, 2024-09-13T15:27:53.082 INFO:tasks.workunit.client.0.smithi112.stdout: "num_flush_mode_high": 0, 2024-09-13T15:27:53.083 INFO:tasks.workunit.client.0.smithi112.stdout: "num_flush_mode_low": 0, 2024-09-13T15:27:53.083 INFO:tasks.workunit.client.0.smithi112.stdout: "num_evict_mode_some": 0, 2024-09-13T15:27:53.083 INFO:tasks.workunit.client.0.smithi112.stdout: "num_evict_mode_full": 0, 2024-09-13T15:27:53.083 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_pinned": 0, 2024-09-13T15:27:53.083 INFO:tasks.workunit.client.0.smithi112.stdout: "num_legacy_snapsets": 0, 2024-09-13T15:27:53.083 INFO:tasks.workunit.client.0.smithi112.stdout: "num_large_omap_objects": 0, 2024-09-13T15:27:53.083 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_manifest": 0, 2024-09-13T15:27:53.083 INFO:tasks.workunit.client.0.smithi112.stdout: "num_omap_bytes": 0, 2024-09-13T15:27:53.083 INFO:tasks.workunit.client.0.smithi112.stdout: "num_omap_keys": 0, 2024-09-13T15:27:53.083 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_repaired": 20 2024-09-13T15:27:53.084 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:27:53.084 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-09-13T15:27:53.084 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-09-13T15:27:53.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: COUNT=20 2024-09-13T15:27:53.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: test 20 = 20 2024-09-13T15:27:53.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-13T15:27:53.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-09-13T15:27:53.515 INFO:tasks.workunit.client.0.smithi112.stderr:dumped all 2024-09-13T15:27:53.532 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:27:53.532 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:27:53.532 INFO:tasks.workunit.client.0.smithi112.stdout: "up_from": 21, 2024-09-13T15:27:53.532 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 90194313225, 2024-09-13T15:27:53.532 INFO:tasks.workunit.client.0.smithi112.stdout: "num_pgs": 1, 2024-09-13T15:27:53.532 INFO:tasks.workunit.client.0.smithi112.stdout: "num_osds": 1, 2024-09-13T15:27:53.532 INFO:tasks.workunit.client.0.smithi112.stdout: "num_per_pool_osds": 1, 2024-09-13T15:27:53.532 INFO:tasks.workunit.client.0.smithi112.stdout: "num_per_pool_omap_osds": 1, 2024-09-13T15:27:53.532 INFO:tasks.workunit.client.0.smithi112.stdout: "kb": 104857600, 2024-09-13T15:27:53.532 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used": 28568, 2024-09-13T15:27:53.532 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_data": 328, 2024-09-13T15:27:53.532 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_omap": 4, 2024-09-13T15:27:53.532 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_meta": 28219, 2024-09-13T15:27:53.533 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_avail": 104829032, 2024-09-13T15:27:53.533 INFO:tasks.workunit.client.0.smithi112.stdout: "statfs": { 2024-09-13T15:27:53.533 INFO:tasks.workunit.client.0.smithi112.stdout: "total": 107374182400, 2024-09-13T15:27:53.533 INFO:tasks.workunit.client.0.smithi112.stdout: "available": 107344928768, 2024-09-13T15:27:53.533 INFO:tasks.workunit.client.0.smithi112.stdout: "internally_reserved": 0, 2024-09-13T15:27:53.533 INFO:tasks.workunit.client.0.smithi112.stdout: "allocated": 335872, 2024-09-13T15:27:53.533 INFO:tasks.workunit.client.0.smithi112.stdout: "data_stored": 55896, 2024-09-13T15:27:53.533 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed": 0, 2024-09-13T15:27:53.533 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed_allocated": 0, 2024-09-13T15:27:53.533 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed_original": 0, 2024-09-13T15:27:53.533 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_allocated": 4398, 2024-09-13T15:27:53.533 INFO:tasks.workunit.client.0.smithi112.stdout: "internal_metadata": 28896978 2024-09-13T15:27:53.533 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:27:53.534 INFO:tasks.workunit.client.0.smithi112.stdout: "hb_peers": [ 2024-09-13T15:27:53.534 INFO:tasks.workunit.client.0.smithi112.stdout: 0 2024-09-13T15:27:53.534 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:27:53.534 INFO:tasks.workunit.client.0.smithi112.stdout: "snap_trim_queue_len": 0, 2024-09-13T15:27:53.534 INFO:tasks.workunit.client.0.smithi112.stdout: "num_snap_trimming": 0, 2024-09-13T15:27:53.534 INFO:tasks.workunit.client.0.smithi112.stdout: "num_shards_repaired": 10, 2024-09-13T15:27:53.534 INFO:tasks.workunit.client.0.smithi112.stdout: "op_queue_age_hist": { 2024-09-13T15:27:53.534 INFO:tasks.workunit.client.0.smithi112.stdout: "histogram": [], 2024-09-13T15:27:53.534 INFO:tasks.workunit.client.0.smithi112.stdout: "upper_bound": 1 2024-09-13T15:27:53.534 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:27:53.534 INFO:tasks.workunit.client.0.smithi112.stdout: "perf_stat": { 2024-09-13T15:27:53.534 INFO:tasks.workunit.client.0.smithi112.stdout: "commit_latency_ms": 0, 2024-09-13T15:27:53.534 INFO:tasks.workunit.client.0.smithi112.stdout: "apply_latency_ms": 0, 2024-09-13T15:27:53.534 INFO:tasks.workunit.client.0.smithi112.stdout: "commit_latency_ns": 0, 2024-09-13T15:27:53.534 INFO:tasks.workunit.client.0.smithi112.stdout: "apply_latency_ns": 0 2024-09-13T15:27:53.535 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:27:53.535 INFO:tasks.workunit.client.0.smithi112.stdout: "alerts": [] 2024-09-13T15:27:53.535 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:27:53.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-09-13T15:27:53.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-13T15:27:53.837 INFO:tasks.workunit.client.0.smithi112.stderr:dumped all 2024-09-13T15:27:53.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: COUNT=10 2024-09-13T15:27:53.855 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: expr 20 / 2 2024-09-13T15:27:53.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: test 10 = 10 2024-09-13T15:27:53.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-13T15:27:53.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-09-13T15:27:54.161 INFO:tasks.workunit.client.0.smithi112.stderr:dumped all 2024-09-13T15:27:54.178 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:27:54.179 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:27:54.179 INFO:tasks.workunit.client.0.smithi112.stdout: "up_from": 24, 2024-09-13T15:27:54.179 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 103079215112, 2024-09-13T15:27:54.179 INFO:tasks.workunit.client.0.smithi112.stdout: "num_pgs": 1, 2024-09-13T15:27:54.179 INFO:tasks.workunit.client.0.smithi112.stdout: "num_osds": 1, 2024-09-13T15:27:54.179 INFO:tasks.workunit.client.0.smithi112.stdout: "num_per_pool_osds": 1, 2024-09-13T15:27:54.179 INFO:tasks.workunit.client.0.smithi112.stdout: "num_per_pool_omap_osds": 1, 2024-09-13T15:27:54.179 INFO:tasks.workunit.client.0.smithi112.stdout: "kb": 104857600, 2024-09-13T15:27:54.180 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used": 28568, 2024-09-13T15:27:54.180 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_data": 328, 2024-09-13T15:27:54.180 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_omap": 4, 2024-09-13T15:27:54.180 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_meta": 28219, 2024-09-13T15:27:54.180 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_avail": 104829032, 2024-09-13T15:27:54.180 INFO:tasks.workunit.client.0.smithi112.stdout: "statfs": { 2024-09-13T15:27:54.180 INFO:tasks.workunit.client.0.smithi112.stdout: "total": 107374182400, 2024-09-13T15:27:54.180 INFO:tasks.workunit.client.0.smithi112.stdout: "available": 107344928768, 2024-09-13T15:27:54.180 INFO:tasks.workunit.client.0.smithi112.stdout: "internally_reserved": 0, 2024-09-13T15:27:54.180 INFO:tasks.workunit.client.0.smithi112.stdout: "allocated": 335872, 2024-09-13T15:27:54.181 INFO:tasks.workunit.client.0.smithi112.stdout: "data_stored": 55896, 2024-09-13T15:27:54.181 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed": 0, 2024-09-13T15:27:54.181 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed_allocated": 0, 2024-09-13T15:27:54.181 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed_original": 0, 2024-09-13T15:27:54.181 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_allocated": 4401, 2024-09-13T15:27:54.181 INFO:tasks.workunit.client.0.smithi112.stdout: "internal_metadata": 28896975 2024-09-13T15:27:54.181 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:27:54.181 INFO:tasks.workunit.client.0.smithi112.stdout: "hb_peers": [ 2024-09-13T15:27:54.181 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:27:54.181 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:27:54.181 INFO:tasks.workunit.client.0.smithi112.stdout: "snap_trim_queue_len": 0, 2024-09-13T15:27:54.181 INFO:tasks.workunit.client.0.smithi112.stdout: "num_snap_trimming": 0, 2024-09-13T15:27:54.181 INFO:tasks.workunit.client.0.smithi112.stdout: "num_shards_repaired": 10, 2024-09-13T15:27:54.181 INFO:tasks.workunit.client.0.smithi112.stdout: "op_queue_age_hist": { 2024-09-13T15:27:54.181 INFO:tasks.workunit.client.0.smithi112.stdout: "histogram": [], 2024-09-13T15:27:54.182 INFO:tasks.workunit.client.0.smithi112.stdout: "upper_bound": 1 2024-09-13T15:27:54.182 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:27:54.182 INFO:tasks.workunit.client.0.smithi112.stdout: "perf_stat": { 2024-09-13T15:27:54.182 INFO:tasks.workunit.client.0.smithi112.stdout: "commit_latency_ms": 0, 2024-09-13T15:27:54.182 INFO:tasks.workunit.client.0.smithi112.stdout: "apply_latency_ms": 0, 2024-09-13T15:27:54.182 INFO:tasks.workunit.client.0.smithi112.stdout: "commit_latency_ns": 0, 2024-09-13T15:27:54.182 INFO:tasks.workunit.client.0.smithi112.stdout: "apply_latency_ns": 0 2024-09-13T15:27:54.182 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:27:54.182 INFO:tasks.workunit.client.0.smithi112.stdout: "alerts": [] 2024-09-13T15:27:54.182 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:27:54.183 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-13T15:27:54.183 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-09-13T15:27:54.473 INFO:tasks.workunit.client.0.smithi112.stderr:dumped all 2024-09-13T15:27:54.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: COUNT=10 2024-09-13T15:27:54.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: expr 20 / 2 2024-09-13T15:27:54.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: test 10 = 10 2024-09-13T15:27:54.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-13T15:27:54.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-09-13T15:27:54.766 INFO:tasks.workunit.client.0.smithi112.stderr:dumped all 2024-09-13T15:27:54.783 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:27:54.783 INFO:tasks.workunit.client.0.smithi112.stdout: "up_from": 0, 2024-09-13T15:27:54.783 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 0, 2024-09-13T15:27:54.783 INFO:tasks.workunit.client.0.smithi112.stdout: "num_pgs": 2, 2024-09-13T15:27:54.783 INFO:tasks.workunit.client.0.smithi112.stdout: "num_osds": 2, 2024-09-13T15:27:54.783 INFO:tasks.workunit.client.0.smithi112.stdout: "num_per_pool_osds": 2, 2024-09-13T15:27:54.783 INFO:tasks.workunit.client.0.smithi112.stdout: "num_per_pool_omap_osds": 2, 2024-09-13T15:27:54.783 INFO:tasks.workunit.client.0.smithi112.stdout: "kb": 209715200, 2024-09-13T15:27:54.783 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used": 57136, 2024-09-13T15:27:54.784 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_data": 656, 2024-09-13T15:27:54.784 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_omap": 8, 2024-09-13T15:27:54.784 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_meta": 56439, 2024-09-13T15:27:54.784 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_avail": 209658064, 2024-09-13T15:27:54.784 INFO:tasks.workunit.client.0.smithi112.stdout: "statfs": { 2024-09-13T15:27:54.784 INFO:tasks.workunit.client.0.smithi112.stdout: "total": 214748364800, 2024-09-13T15:27:54.784 INFO:tasks.workunit.client.0.smithi112.stdout: "available": 214689857536, 2024-09-13T15:27:54.784 INFO:tasks.workunit.client.0.smithi112.stdout: "internally_reserved": 0, 2024-09-13T15:27:54.784 INFO:tasks.workunit.client.0.smithi112.stdout: "allocated": 671744, 2024-09-13T15:27:54.784 INFO:tasks.workunit.client.0.smithi112.stdout: "data_stored": 111792, 2024-09-13T15:27:54.784 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed": 0, 2024-09-13T15:27:54.784 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed_allocated": 0, 2024-09-13T15:27:54.784 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed_original": 0, 2024-09-13T15:27:54.784 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_allocated": 8799, 2024-09-13T15:27:54.785 INFO:tasks.workunit.client.0.smithi112.stdout: "internal_metadata": 57793953 2024-09-13T15:27:54.785 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:27:54.785 INFO:tasks.workunit.client.0.smithi112.stdout: "hb_peers": [], 2024-09-13T15:27:54.785 INFO:tasks.workunit.client.0.smithi112.stdout: "snap_trim_queue_len": 0, 2024-09-13T15:27:54.785 INFO:tasks.workunit.client.0.smithi112.stdout: "num_snap_trimming": 0, 2024-09-13T15:27:54.785 INFO:tasks.workunit.client.0.smithi112.stdout: "num_shards_repaired": 20, 2024-09-13T15:27:54.785 INFO:tasks.workunit.client.0.smithi112.stdout: "op_queue_age_hist": { 2024-09-13T15:27:54.785 INFO:tasks.workunit.client.0.smithi112.stdout: "histogram": [], 2024-09-13T15:27:54.785 INFO:tasks.workunit.client.0.smithi112.stdout: "upper_bound": 1 2024-09-13T15:27:54.786 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:27:54.786 INFO:tasks.workunit.client.0.smithi112.stdout: "perf_stat": { 2024-09-13T15:27:54.786 INFO:tasks.workunit.client.0.smithi112.stdout: "commit_latency_ms": 0, 2024-09-13T15:27:54.786 INFO:tasks.workunit.client.0.smithi112.stdout: "apply_latency_ms": 0, 2024-09-13T15:27:54.786 INFO:tasks.workunit.client.0.smithi112.stdout: "commit_latency_ns": 0, 2024-09-13T15:27:54.786 INFO:tasks.workunit.client.0.smithi112.stdout: "apply_latency_ns": 0 2024-09-13T15:27:54.786 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:27:54.786 INFO:tasks.workunit.client.0.smithi112.stdout: "alerts": [], 2024-09-13T15:27:54.786 INFO:tasks.workunit.client.0.smithi112.stdout: "network_ping_times": [] 2024-09-13T15:27:54.786 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:27:54.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-09-13T15:27:54.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-13T15:27:55.096 INFO:tasks.workunit.client.0.smithi112.stderr:dumped all 2024-09-13T15:27:55.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: COUNT=20 2024-09-13T15:27:55.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: test 20 = 20 2024-09-13T15:27:55.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T15:27:55.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:27:55.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:27:55.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:27:55.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:27:55.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:27:55.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:27:55.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:27:55.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:27:55.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:27:55.224 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:27:55.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:27:55.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:27:55.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:27:55.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:27:55.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:27:55.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:27:55.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:27:55.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:27:55.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:27:55.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:27:55.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:27:55.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:27:55.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:27:55.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:27:55.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:27:55.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:27:55.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:27:55.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:27:55.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T15:27:55.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T15:27:55.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T15:27:55.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T15:27:55.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:27:55.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:27:55.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:27:55.248 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:27:55.248 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:27:55.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:27:55.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:27:55.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:27:55.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:27:55.250 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:27:55.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:27:55.251 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:27:55.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:27:55.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:27:55.252 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:27:55.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:27:55.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:27:55.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:27:55.254 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:27:55.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:27:55.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:27:55.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:27:55.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:27:55.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:27:55.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:27:55.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:27:55.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:27:55.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:27:55.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T15:27:55.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:27:55.260 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:27:55.260 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:27:55.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T15:27:55.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:27:55.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T15:27:55.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:27:55.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T15:27:55.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-09-13T15:27:55.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-09-13T15:27:55.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local poolname=testpool 2024-09-13T15:27:55.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local OSDS=3 2024-09-13T15:27:55.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OBJS=30 2024-09-13T15:27:55.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats_ec: local REPAIRS=26 2024-09-13T15:27:55.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local allow_overwrites=false 2024-09-13T15:27:55.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-09-13T15:27:55.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T15:27:55.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:27:55.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:27:55.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:27:55.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T15:27:55.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T15:27:55.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:27:55.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:27:55.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:27:55.802 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:27:55.803 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:27:55.803 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:27:55.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:27:55.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T15:27:56.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:27:56.058 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:27:56.058 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:27:56.058 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:27:56.059 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:27:56.059 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:27:56.059 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:27:56.059 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:27:56.059 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:27:56.059 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:27:56.060 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:27:56.060 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:27:56.060 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:27:56.060 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:27:56.060 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T15:27:56.174 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:27:56.174 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:27:56.174 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:27:56.174 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:27:56.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:27:56.176 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:27:56.176 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:27:56.176 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:27:56.176 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:27:56.176 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:27:56.176 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:27:56.177 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:27:56.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:27:56.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T15:27:56.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-09-13T15:27:56.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T15:27:56.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:27:56.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:27:56.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:27:56.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T15:27:56.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:27:57.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:27:57.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:27:57.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:27:57.979 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:27:57.979 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:27:57.979 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:27:57.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:27:57.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:27:57.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:27:58.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T15:27:58.020 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: expr 3 - 1 2024-09-13T15:27:58.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: seq 0 2 2024-09-13T15:27:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:27:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:27:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:27:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:27:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:27:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:27:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:27:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:27:58.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:27:58.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:27:58.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:27:58.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:27:58.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:27:58.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:27:58.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:27:58.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:27:58.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:27:58.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:27:58.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:27:58.025 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:27:58.026 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:27:58.026 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:27:58.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:27:58.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:27:58.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:27:58.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:27:58.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:27:58.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:27:58.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:27:58.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:27:58.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:27:58.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:27:58.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:27:58.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:27:58.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T15:27:58.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:27:58.030 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:27:58.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b43d1f62-e4b2-4c62-ac50-fc1e206911c9 2024-09-13T15:27:58.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b43d1f62-e4b2-4c62-ac50-fc1e206911c9' 2024-09-13T15:27:58.031 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 b43d1f62-e4b2-4c62-ac50-fc1e206911c9 2024-09-13T15:27:58.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:27:58.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD+WeRm14rGAhAAZb9HjqFuI3KnrLFT407JBA== 2024-09-13T15:27:58.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD+WeRm14rGAhAAZb9HjqFuI3KnrLFT407JBA=="}' 2024-09-13T15:27:58.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b43d1f62-e4b2-4c62-ac50-fc1e206911c9 -i td/osd-scrub-repair/0/new.json 2024-09-13T15:27:58.357 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:27:58.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T15:27:58.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQD+WeRm14rGAhAAZb9HjqFuI3KnrLFT407JBA== --osd-uuid b43d1f62-e4b2-4c62-ac50-fc1e206911c9 2024-09-13T15:27:58.423 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:27:58.423+0000 7f4bd93527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:27:58.517 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:27:58.515+0000 7f4bd93527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:27:58.519 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:27:58.519+0000 7f4bd93527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:27:58.519 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:27:58.519+0000 7f4bd93527c0 -1 bdev(0x557332fddc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:27:58.519 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:27:58.519+0000 7f4bd93527c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T15:28:06.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T15:28:06.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:28:06.431 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:28:06.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:28:06.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:28:06.828 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:28:06.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:28:06.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:28:06.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:28:06.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:28:06.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:28:06.863 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:28:06.863+0000 7f44290a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:28:06.897 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:28:06.895+0000 7f44290a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:28:06.898 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:28:06.899+0000 7f44290a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:28:07.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:28:07.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:28:07.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:28:07.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:28:07.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:28:07.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:07.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:28:07.204 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:28:07.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:07.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:07.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:07.961 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:28:07.959+0000 7f44290a57c0 -1 Falling back to public interface 2024-09-13T15:28:08.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:08.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:08.513 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:28:08.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:28:08.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:08.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:08.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:09.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:09.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:09.852 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:28:09.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:28:09.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:09.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:10.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:11.042 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:28:11.039+0000 7f44290a57c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:28:11.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:11.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:11.182 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:28:11.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:28:11.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:11.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:11.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:12.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:12.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:12.684 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:28:12.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:28:12.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:12.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:13.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:14.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:14.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:14.121 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:28:14.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:28:14.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:14.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:14.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:15.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:15.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:15.461 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:28:15.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:28:15.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:15.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:15.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:16.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:16.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:16.939 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:28:16.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:28:16.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:16.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:17.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:18.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:18.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:18.275 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:28:18.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:28:18.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:18.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:18.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:19.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:19.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:19.603 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:28:19.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:28:19.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:19.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:20.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:21.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:21.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:21.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:28:21.162 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:28:21.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:21.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:21.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:22.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:22.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:22.482 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:28:22.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:28:22.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:22.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:22.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:23.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:23.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:23.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:28:23.859 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:28:23.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:23.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:24.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:25.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:25.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:25.267 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:28:25.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:28:25.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:25.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:25.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:26.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:26.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:26.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:28:26.827 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:28:26.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:26.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:27.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:28.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:28.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:28.469 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:28:28.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:28:28.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:28.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:28.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:29.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:29.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:29.822 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:28:29.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:28:29.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:29.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:30.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:31.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:31.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:31.548 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:28:31.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:28:31.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:31.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:32.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:33.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:33.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:33.168 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:28:33.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:28:33.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:33.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:34.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:35.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:35.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:35.195 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:28:35.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:28:35.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:35.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:35.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:36.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:36.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:36.954 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:28:36.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:28:36.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:36.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:37.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:38.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:38.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:38.628 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:28:38.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:28:38.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:38.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:39.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:40.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:40.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:40.329 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:28:40.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:28:40.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:40.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:41.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:42.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:42.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:42.438 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:28:42.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:28:42.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:42.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:42.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:43.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:43.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:43.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:28:43.931 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:28:43.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:43.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:28:44.301 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3917888217,v1:127.0.0.1:6803/3917888217] [v2:127.0.0.1:6804/3917888217,v1:127.0.0.1:6805/3917888217] exists,up b43d1f62-e4b2-4c62-ac50-fc1e206911c9 2024-09-13T15:28:44.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:28:44.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:28:44.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:28:44.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:28:44.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:28:44.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:28:44.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:28:44.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T15:28:44.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:28:44.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:28:44.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:28:44.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:28:44.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:28:44.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:28:44.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:28:44.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:28:44.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:28:44.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:28:44.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:28:44.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:28:44.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:28:44.311 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:28:44.311 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:28:44.311 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:28:44.311 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:28:44.311 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:28:44.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:28:44.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:28:44.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:28:44.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:28:44.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:28:44.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:28:44.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:28:44.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:28:44.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:28:44.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:28:44.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:28:44.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T15:28:44.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:28:44.313 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:28:44.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=89a47ead-773c-4e56-869b-c532e71701f4 2024-09-13T15:28:44.314 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 89a47ead-773c-4e56-869b-c532e71701f4 2024-09-13T15:28:44.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 89a47ead-773c-4e56-869b-c532e71701f4' 2024-09-13T15:28:44.315 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:28:44.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAsWuRmHUCHExAAMGijzkqfqLC1dMJ0tD7ZKg== 2024-09-13T15:28:44.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAsWuRmHUCHExAAMGijzkqfqLC1dMJ0tD7ZKg=="}' 2024-09-13T15:28:44.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 89a47ead-773c-4e56-869b-c532e71701f4 -i td/osd-scrub-repair/1/new.json 2024-09-13T15:28:44.899 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:28:44.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T15:28:44.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAsWuRmHUCHExAAMGijzkqfqLC1dMJ0tD7ZKg== --osd-uuid 89a47ead-773c-4e56-869b-c532e71701f4 2024-09-13T15:28:44.961 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:28:44.958+0000 7fb8a59287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:28:45.015 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:28:45.014+0000 7fb8a59287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:28:45.016 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:28:45.014+0000 7fb8a59287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:28:45.017 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:28:45.014+0000 7fb8a59287c0 -1 bdev(0x55f5d67cdc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T15:28:45.017 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:28:45.014+0000 7fb8a59287c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T15:28:53.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T15:28:53.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:28:53.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T15:28:53.673 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T15:28:53.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:28:54.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T15:28:54.084 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:28:54.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:28:54.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:28:54.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:28:54.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:28:54.117 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:28:54.114+0000 7f040d4b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:28:54.161 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:28:54.158+0000 7f040d4b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:28:54.163 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:28:54.162+0000 7f040d4b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:28:54.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T15:28:54.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:28:54.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:28:54.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:28:54.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:28:54.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:54.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:28:54.390 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:28:54.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:54.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:28:54.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:55.765 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:28:55.762+0000 7f040d4b97c0 -1 Falling back to public interface 2024-09-13T15:28:55.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:55.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:55.776 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:28:55.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:28:55.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:55.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:28:56.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:57.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:57.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:57.096 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:28:57.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:28:57.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:57.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:28:57.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:58.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:58.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:58.434 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:28:58.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:28:58.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:58.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:28:58.719 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:28:58.718+0000 7f040d4b97c0 -1 osd.1 0 log_to_monitors true 2024-09-13T15:28:58.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:28:59.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:28:59.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:28:59.773 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:28:59.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:28:59.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:28:59.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:00.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:01.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:01.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:01.244 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:29:01.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:29:01.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:01.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:01.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:02.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:02.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:02.620 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:29:02.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:29:02.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:02.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:03.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:04.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:04.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:04.113 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:29:04.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:29:04.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:04.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:04.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:05.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:05.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:05.450 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:29:05.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:29:05.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:05.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:05.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:06.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:06.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:06.990 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:29:06.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:29:06.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:06.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:07.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:08.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:08.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:08.323 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:29:08.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:29:08.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:08.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:08.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:09.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:09.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:09.762 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:29:09.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:29:09.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:09.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:10.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:11.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:11.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:11.298 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:29:11.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:29:11.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:11.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:11.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:12.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:12.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:12.752 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:29:12.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:29:12.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:12.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:13.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:14.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:14.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:14.319 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:29:14.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:29:14.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:14.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:14.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:15.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:15.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:15.733 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:29:15.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:29:15.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:15.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:16.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:17.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:17.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:17.072 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:29:17.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:29:17.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:17.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:17.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:18.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:18.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:18.413 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:29:18.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:29:18.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:18.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:18.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:19.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:19.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:19.842 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:29:19.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:29:19.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:19.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:20.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:21.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:21.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:21.513 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:29:21.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:29:21.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:21.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:22.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:23.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:23.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:23.445 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:29:23.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:29:23.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:23.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:24.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:25.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:25.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:25.329 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:29:25.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:29:25.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:25.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:26.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:27.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:27.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:27.127 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:29:27.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:29:27.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:27.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:28.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:29.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:29.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:29.149 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:29:29.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:29:29.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:29.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:29.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:30.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:30.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:30.783 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:29:30.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:29:30.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:30.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:31.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:32.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:32.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:32.425 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:29:32.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:29:32.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:32.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:29:32.754 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2835109236,v1:127.0.0.1:6811/2835109236] [v2:127.0.0.1:6812/2835109236,v1:127.0.0.1:6813/2835109236] exists,up 89a47ead-773c-4e56-869b-c532e71701f4 2024-09-13T15:29:32.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:29:32.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:29:32.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:29:32.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:29:32.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:29:32.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:29:32.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:29:32.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T15:29:32.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:29:32.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:29:32.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:29:32.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:29:32.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:29:32.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:29:32.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T15:29:32.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T15:29:32.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:29:32.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:29:32.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:29:32.757 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:29:32.757 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:29:32.757 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:29:32.757 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:29:32.757 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:29:32.757 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:29:32.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:29:32.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:29:32.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:29:32.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:29:32.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:29:32.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:29:32.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:29:32.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:29:32.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:29:32.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:29:32.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:29:32.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:29:32.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T15:29:32.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T15:29:32.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:29:32.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7351a202-895e-490c-a308-14910c583381 2024-09-13T15:29:32.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 7351a202-895e-490c-a308-14910c583381' 2024-09-13T15:29:32.761 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 7351a202-895e-490c-a308-14910c583381 2024-09-13T15:29:32.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:29:32.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBcWuRmyWCqLhAAg1AbBkV1rGLqqWmO6OFTbQ== 2024-09-13T15:29:32.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBcWuRmyWCqLhAAg1AbBkV1rGLqqWmO6OFTbQ=="}' 2024-09-13T15:29:32.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7351a202-895e-490c-a308-14910c583381 -i td/osd-scrub-repair/2/new.json 2024-09-13T15:29:33.364 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:29:33.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T15:29:33.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBcWuRmyWCqLhAAg1AbBkV1rGLqqWmO6OFTbQ== --osd-uuid 7351a202-895e-490c-a308-14910c583381 2024-09-13T15:29:33.421 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:29:33.417+0000 7f5faa9367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:29:33.498 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:29:33.493+0000 7f5faa9367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:29:33.500 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:29:33.497+0000 7f5faa9367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:29:33.500 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:29:33.497+0000 7f5faa9367c0 -1 bdev(0x55eaf8586800 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T15:29:33.500 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:29:33.497+0000 7f5faa9367c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T15:29:42.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T15:29:42.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:29:42.342 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T15:29:42.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T15:29:42.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:29:42.755 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T15:29:42.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T15:29:42.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:29:42.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:29:42.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:29:42.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:29:42.791 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:29:42.788+0000 7f59ed0377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:29:42.819 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:29:42.816+0000 7f59ed0377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:29:42.821 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:29:42.816+0000 7f59ed0377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:29:43.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T15:29:43.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:29:43.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:29:43.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:29:43.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:29:43.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:43.118 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:29:43.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:29:43.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:43.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:29:43.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:44.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:44.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:44.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:29:44.439 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:29:44.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:44.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:29:44.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:44.808 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:29:44.804+0000 7f59ed0377c0 -1 Falling back to public interface 2024-09-13T15:29:45.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:45.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:45.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:29:45.762 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:29:45.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:45.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:29:46.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:47.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:47.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:47.087 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:29:47.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:29:47.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:47.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:29:47.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:47.962 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:29:47.956+0000 7f59ed0377c0 -1 osd.2 0 log_to_monitors true 2024-09-13T15:29:48.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:48.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:48.404 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:29:48.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:29:48.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:48.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:29:49.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:50.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:50.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:50.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:29:50.060 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:29:50.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:50.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:29:50.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:51.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:51.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:51.459 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:29:51.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:29:51.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:51.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:29:52.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:53.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:53.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:53.011 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:29:53.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:29:53.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:53.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:29:53.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:54.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:54.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:54.475 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:29:54.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:29:54.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:54.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:29:54.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:55.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:55.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:55.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:29:55.861 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:29:55.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:55.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:29:56.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:57.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:57.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:57.234 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:29:57.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:29:57.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:57.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:29:57.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:29:58.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:29:58.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:29:58.661 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:29:58.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:29:58.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:29:58.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:29:59.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:30:00.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:30:00.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:30:00.274 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:30:00.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:30:00.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:30:00.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:30:00.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:30:01.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:30:01.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:30:01.613 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:30:01.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:30:01.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:30:01.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:30:01.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:30:02.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:30:02.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:30:02.976 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:30:02.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:30:02.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:30:02.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:30:03.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:30:04.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:30:04.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:30:04.498 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:30:04.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:30:04.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:30:04.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:30:04.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:30:05.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:30:05.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:30:05.877 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:30:05.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:30:05.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:30:05.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:30:06.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:30:07.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:30:07.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:30:07.410 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:30:07.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:30:07.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:30:07.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:30:08.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:30:09.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:30:09.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:30:09.004 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:30:09.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:30:09.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:30:09.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:30:10.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:30:11.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:30:11.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:30:11.077 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:30:11.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:30:11.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:30:11.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:30:12.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:30:13.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:30:13.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:30:13.122 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:30:13.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:30:13.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:30:13.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:30:14.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:30:15.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:30:15.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:30:15.120 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:30:15.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:30:15.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:30:15.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:30:15.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:30:16.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:30:16.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:30:16.899 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:30:16.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:30:16.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:30:16.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:30:17.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:30:18.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:30:18.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:30:18.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:30:18.717 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:30:18.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:30:18.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:30:19.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:30:20.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:30:20.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:30:20.170 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:30:20.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:30:20.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:30:20.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:30:20.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:30:21.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:30:21.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:30:21.606 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:30:21.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:30:21.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:30:21.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:30:21.941 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4284950907,v1:127.0.0.1:6819/4284950907] [v2:127.0.0.1:6820/4284950907,v1:127.0.0.1:6821/4284950907] exists,up 7351a202-895e-490c-a308-14910c583381 2024-09-13T15:30:21.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:30:21.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:30:21.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:30:21.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-09-13T15:30:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=testpool 2024-09-13T15:30:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-13T15:30:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-13T15:30:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-13T15:30:21.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-13T15:30:22.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-09-13T15:30:22.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-09-13T15:30:23.461 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'testpool' created 2024-09-13T15:30:23.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:30:24.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-13T15:30:24.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-13T15:30:24.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:30:24.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:30:24.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:30:24.488 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:30:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:30:24.489 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:30:24.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:30:24.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:30:24.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:30:24.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:30:24.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:30:24.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:30:24.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:30:24.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:30:24.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:30:24.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:30:24.980 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:30:24.980 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:30:24.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:30:24.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:30:24.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:30:25.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-13T15:30:25.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-13T15:30:25.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-13T15:30:25.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:30:25.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:30:25.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-13T15:30:25.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-13T15:30:25.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672979' 2024-09-13T15:30:25.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:30:25.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:30:25.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-13T15:30:25.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-13T15:30:25.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672979 2-64424509451' 2024-09-13T15:30:25.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:30:25.459 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-13T15:30:25.460 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:30:25.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:30:25.461 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-13T15:30:25.461 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:30:25.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-13T15:30:25.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-13T15:30:25.463 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836504 2024-09-13T15:30:25.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:30:25.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836504 2024-09-13T15:30:25.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:30:26.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:30:26.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:30:27.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-13T15:30:27.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:30:27.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-13T15:30:27.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:30:27.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:30:27.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-13T15:30:27.219 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:30:27.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-13T15:30:27.220 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672979 2024-09-13T15:30:27.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-13T15:30:27.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:30:27.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-13T15:30:27.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:30:27.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-13T15:30:27.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:30:27.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:30:27.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-13T15:30:27.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:30:27.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-13T15:30:27.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-13T15:30:27.538 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509451 2024-09-13T15:30:27.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:30:27.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509451 2024-09-13T15:30:27.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:30:27.862 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:30:27.862 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:30:28.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:30:28.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:30:28.251 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:30:28.251 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:30:28.251 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:30:28.251 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:30:28.252 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:30:28.252 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:30:28.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T15:30:28.634 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:30:28.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:30:28.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:30:29.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T15:30:29.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T15:30:29.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:30:29.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T15:30:29.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:30:29.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:30:29.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:30:29.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:30:29.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:30:29.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:30:29.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:30:29.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:30:29.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:30:29.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:30:29.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:30:29.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:30:29.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:30:29.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:30:29.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:30:29.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:30:29.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:30:29.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-13T15:30:29.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats_ec: local payload=ABCDEF 2024-09-13T15:30:29.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: echo ABCDEF 2024-09-13T15:30:29.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: seq 1 30 2024-09-13T15:30:29.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:29.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:30.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:30.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:30.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:30.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:30.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:30.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:31.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:31.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:31.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:31.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:31.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:31.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:32.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:32.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:32.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:32.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:32.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:32.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:32.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:32.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:33.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:33.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:33.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:33.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:33.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:33.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:34.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:34.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:34.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:34.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:34.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:34.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:35.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:35.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:35.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:35.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:35.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:35.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:36.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:36.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:36.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:36.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:36.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:36.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:37.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:37.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:37.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:37.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:37.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:37.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:37.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:37.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:38.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:38.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:38.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:38.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:38.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-09-13T15:30:38.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-09-13T15:30:39.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-09-13T15:30:39.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-13T15:30:39.302 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-13T15:30:39.302 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2024-09-13T15:30:39.302 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T15:30:39.302 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T15:30:39.302 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:30:39.302 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-13T15:30:39.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T15:30:39.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2024-09-13T15:30:39.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T15:30:40.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: local other=0 2024-09-13T15:30:40.039 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_pg testpool obj1 2024-09-13T15:30:40.039 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-13T15:30:40.039 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2024-09-13T15:30:40.039 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2024-09-13T15:30:40.039 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T15:30:40.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local pgid=1.0 2024-09-13T15:30:40.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_primary testpool obj1 2024-09-13T15:30:40.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-13T15:30:40.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T15:30:40.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-13T15:30:40.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:30:40.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local primary=1 2024-09-13T15:30:40.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:758: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:30:40.683 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:30:40.683 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:30:40.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:30:40.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:30:40.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:30:41.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:30:41.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-13T15:30:41.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:30:41.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:30:41.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:30:41.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:30:41.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:30:43.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:30:43.315 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: seq 1 26 2024-09-13T15:30:43.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:30:43.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 1 % 2 2024-09-13T15:30:43.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-13T15:30:43.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-09-13T15:30:43.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:30:43.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:30:43.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:30:43.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:30:43.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:30:43.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-09-13T15:30:45.464 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-09-13T15:30:45.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:30:45.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 2 % 2 2024-09-13T15:30:45.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-13T15:30:45.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-13T15:30:45.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:30:45.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:30:45.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:30:45.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:30:45.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:30:46.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-13T15:30:48.264 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#1:104778fc:::obj2:head# 2024-09-13T15:30:48.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:30:48.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 3 % 2 2024-09-13T15:30:48.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-13T15:30:48.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-09-13T15:30:48.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:30:48.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:30:48.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:30:48.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:30:48.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:30:48.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-09-13T15:30:50.240 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-09-13T15:30:50.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:30:50.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 4 % 2 2024-09-13T15:30:50.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-13T15:30:50.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-09-13T15:30:50.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:30:50.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:30:50.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:30:50.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:30:50.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:30:50.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-09-13T15:30:52.235 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-09-13T15:30:52.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:30:52.769 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 5 % 2 2024-09-13T15:30:52.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-13T15:30:52.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-09-13T15:30:52.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:30:52.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:30:52.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:30:52.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:30:52.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:30:52.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-09-13T15:30:54.137 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#1:c52c9666:::obj5:head# 2024-09-13T15:30:54.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:30:54.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 6 % 2 2024-09-13T15:30:54.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-13T15:30:54.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-09-13T15:30:54.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:30:54.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:30:54.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:30:54.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:30:54.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:30:54.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-09-13T15:30:55.965 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#1:233a42c1:::obj6:head# 2024-09-13T15:30:56.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:30:56.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 7 % 2 2024-09-13T15:30:56.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-13T15:30:56.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-09-13T15:30:56.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:30:56.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:30:56.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:30:56.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:30:56.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:30:56.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-09-13T15:30:58.026 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#1:9a09113a:::obj7:head# 2024-09-13T15:30:58.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:30:58.561 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 8 % 2 2024-09-13T15:30:58.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-13T15:30:58.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-09-13T15:30:58.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:30:58.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:30:58.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:30:58.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:30:58.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:30:58.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-09-13T15:30:59.795 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#1:a557efb1:::obj8:head# 2024-09-13T15:31:00.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:00.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 9 % 2 2024-09-13T15:31:00.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-13T15:31:00.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-09-13T15:31:00.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:00.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:00.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:31:00.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:00.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:31:00.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-09-13T15:31:01.639 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#1:fe351e27:::obj9:head# 2024-09-13T15:31:02.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:02.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 10 % 2 2024-09-13T15:31:02.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-13T15:31:02.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-09-13T15:31:02.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:02.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:02.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:31:02.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:02.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:31:02.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-09-13T15:31:03.550 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#1:d1337354:::obj10:head# 2024-09-13T15:31:04.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:04.086 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 11 % 2 2024-09-13T15:31:04.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-13T15:31:04.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-09-13T15:31:04.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:04.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:04.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:31:04.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:04.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:31:04.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-09-13T15:31:05.477 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-09-13T15:31:06.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:06.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 12 % 2 2024-09-13T15:31:06.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-13T15:31:06.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-09-13T15:31:06.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:06.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:06.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:31:06.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:06.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:31:06.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-09-13T15:31:07.790 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-09-13T15:31:08.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:08.830 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 13 % 2 2024-09-13T15:31:08.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-13T15:31:08.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-09-13T15:31:08.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:08.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:08.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:31:08.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:08.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:31:08.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-09-13T15:31:09.977 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-09-13T15:31:10.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:10.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 14 % 2 2024-09-13T15:31:10.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-13T15:31:10.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-09-13T15:31:10.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:10.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:10.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:31:10.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:10.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:31:10.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-09-13T15:31:11.737 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#1:e0a44829:::obj14:head# 2024-09-13T15:31:12.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:12.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 15 % 2 2024-09-13T15:31:12.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-13T15:31:12.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-09-13T15:31:12.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:12.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:12.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:31:12.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:12.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:31:12.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-09-13T15:31:13.539 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#1:ab946124:::obj15:head# 2024-09-13T15:31:14.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:14.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 16 % 2 2024-09-13T15:31:14.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-13T15:31:14.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-09-13T15:31:14.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:14.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:14.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:31:14.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:14.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:31:14.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-09-13T15:31:15.450 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-09-13T15:31:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:15.993 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 17 % 2 2024-09-13T15:31:15.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-13T15:31:15.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-09-13T15:31:15.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:15.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:15.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:31:15.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:15.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:31:15.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-09-13T15:31:17.360 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-09-13T15:31:17.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:17.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 18 % 2 2024-09-13T15:31:17.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-13T15:31:17.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-09-13T15:31:17.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:17.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:17.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:31:17.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:17.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:31:17.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-09-13T15:31:19.278 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#1:b9836a99:::obj18:head# 2024-09-13T15:31:19.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:19.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 19 % 2 2024-09-13T15:31:19.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-13T15:31:19.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-09-13T15:31:19.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:19.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:19.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:31:19.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:19.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:31:19.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-09-13T15:31:21.081 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-09-13T15:31:21.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:21.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 20 % 2 2024-09-13T15:31:21.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-13T15:31:21.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-09-13T15:31:21.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:21.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:21.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:31:21.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:21.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:31:21.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-09-13T15:31:22.775 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#1:24b5611b:::obj20:head# 2024-09-13T15:31:23.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:23.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 21 % 2 2024-09-13T15:31:23.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-13T15:31:23.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-09-13T15:31:23.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:23.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:23.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:31:23.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:23.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:31:23.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-09-13T15:31:24.561 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-09-13T15:31:25.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:25.102 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 22 % 2 2024-09-13T15:31:25.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-13T15:31:25.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-09-13T15:31:25.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:25.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:25.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:31:25.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:25.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:31:25.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-09-13T15:31:26.413 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#1:c0c49def:::obj22:head# 2024-09-13T15:31:26.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:26.961 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 23 % 2 2024-09-13T15:31:26.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-13T15:31:26.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-09-13T15:31:26.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:26.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:26.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:31:26.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:26.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:31:26.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-09-13T15:31:28.249 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#1:242c40e0:::obj23:head# 2024-09-13T15:31:28.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:28.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 24 % 2 2024-09-13T15:31:28.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-13T15:31:28.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-09-13T15:31:28.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:28.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:28.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:31:28.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:28.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:31:28.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-09-13T15:31:30.119 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#1:7eacede5:::obj24:head# 2024-09-13T15:31:30.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:30.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 25 % 2 2024-09-13T15:31:30.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-09-13T15:31:30.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-09-13T15:31:30.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:30.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:30.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T15:31:30.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:30.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:31:30.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-09-13T15:31:31.930 INFO:tasks.workunit.client.0.smithi112.stdout:remove 0#1:7109478f:::obj25:head# 2024-09-13T15:31:32.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-09-13T15:31:32.466 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 26 % 2 2024-09-13T15:31:32.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-09-13T15:31:32.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-09-13T15:31:32.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:31:32.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:31:32.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:31:32.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:31:32.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:31:32.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-09-13T15:31:33.799 INFO:tasks.workunit.client.0.smithi112.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-09-13T15:31:34.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:31:34.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:31:34.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:31:34.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:31:34.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:31:34.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:31:34.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:31:34.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:31:34.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:31:34.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:31:34.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:31:34.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:31:34.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:31:34.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:31:34.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:31:34.335 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:31:34.335 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:31:34.336 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:31:34.336 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:31:34.336 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:31:34.336 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:31:34.336 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:31:34.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:31:34.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:31:34.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:31:34.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:31:34.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:31:34.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:31:34.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:31:34.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:31:34.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:31:34.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T15:31:34.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:31:34.338 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:31:34.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:31:34.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:31:34.339 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-13T15:31:34.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:31:34.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:31:34.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:31:34.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:31:34.376 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:31:34.374+0000 7f563db617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:31:34.379 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:31:34.378+0000 7f563db617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:31:34.380 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:31:34.378+0000 7f563db617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:31:34.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:31:34.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:31:34.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:31:34.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:31:34.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:31:34.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:31:34.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:31:34.674 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:31:34.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:31:34.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:31:35.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:31:35.661 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:31:35.657+0000 7f563db617c0 -1 Falling back to public interface 2024-09-13T15:31:36.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:31:36.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:31:36.003 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:31:36.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:31:36.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:31:36.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:31:36.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:31:37.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:31:37.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:31:37.323 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:31:37.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:31:37.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:31:37.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:31:37.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:31:38.562 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:31:38.561+0000 7f563db617c0 -1 osd.1 23 log_to_monitors true 2024-09-13T15:31:38.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:31:38.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:31:38.649 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:31:38.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:31:38.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:31:38.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:31:39.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:31:40.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:31:40.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:31:40.075 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:31:40.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:31:40.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:31:40.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:31:40.436 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 27 up_thru 27 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6802/2755479006,v1:127.0.0.1:6803/2755479006] [v2:127.0.0.1:6804/2755479006,v1:127.0.0.1:6805/2755479006] exists,up 89a47ead-773c-4e56-869b-c532e71701f4 2024-09-13T15:31:40.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:31:40.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:31:40.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:31:40.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:31:40.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:31:40.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:31:40.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:31:40.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:31:40.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:31:40.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:31:40.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:31:40.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:31:40.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:31:40.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:31:40.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:31:40.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:31:40.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:31:40.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:31:40.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:31:40.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:31:40.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:31:40.440 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:31:40.441 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:31:40.441 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:31:40.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:31:40.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:31:40.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:31:40.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:31:40.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:31:40.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:31:40.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:31:40.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:31:40.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:31:40.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:31:40.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-13T15:31:40.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:31:40.442 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:31:40.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:31:40.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-13T15:31:40.442 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:31:40.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:31:40.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:31:40.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:31:40.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:31:40.481 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:31:40.477+0000 7f5a8f2aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:31:40.517 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:31:40.513+0000 7f5a8f2aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:31:40.519 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:31:40.517+0000 7f5a8f2aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:31:40.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:31:40.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:31:40.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:31:40.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:31:40.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:31:40.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:31:40.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:31:40.772 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:31:40.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:31:40.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:31:41.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:31:41.809 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:31:41.805+0000 7f5a8f2aa7c0 -1 Falling back to public interface 2024-09-13T15:31:42.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:31:42.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:31:42.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:31:42.180 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:31:42.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:31:42.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:31:42.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:31:43.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:31:43.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:31:43.499 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:31:43.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:31:43.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:31:43.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:31:43.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:31:44.718 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:31:44.713+0000 7f5a8f2aa7c0 -1 osd.0 21 log_to_monitors true 2024-09-13T15:31:44.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:31:44.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:31:44.836 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:31:44.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:31:44.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:31:44.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:31:45.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:31:46.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:31:46.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:31:46.344 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:31:46.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:31:46.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:31:46.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:31:46.697 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/1708766512,v1:127.0.0.1:6811/1708766512] [v2:127.0.0.1:6812/1708766512,v1:127.0.0.1:6813/1708766512] exists,up b43d1f62-e4b2-4c62-ac50-fc1e206911c9 2024-09-13T15:31:46.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:31:46.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:31:46.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:31:46.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2024-09-13T15:31:46.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:31:46.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:31:46.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:31:46.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:31:46.700 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:31:46.700 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:31:46.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:31:46.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:31:46.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:31:46.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:31:46.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:31:46.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:31:46.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:31:46.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:31:46.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:31:47.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:31:47.159 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:31:47.160 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:31:47.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:31:47.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:31:47.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:31:47.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018882 2024-09-13T15:31:47.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018882 2024-09-13T15:31:47.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882' 2024-09-13T15:31:47.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:31:47.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:31:47.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964116995 2024-09-13T15:31:47.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964116995 2024-09-13T15:31:47.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-115964116995' 2024-09-13T15:31:47.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:31:47.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:31:47.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509469 2024-09-13T15:31:47.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509469 2024-09-13T15:31:47.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-115964116995 2-64424509469' 2024-09-13T15:31:47.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:31:47.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018882 2024-09-13T15:31:47.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:31:47.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:31:47.581 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018882 2024-09-13T15:31:47.581 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:31:47.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018882 2024-09-13T15:31:47.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-09-13T15:31:47.582 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 128849018882 2024-09-13T15:31:47.583 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:31:47.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 128849018882 2024-09-13T15:31:47.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:31:48.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:31:48.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:31:49.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-13T15:31:49.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:31:49.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-115964116995 2024-09-13T15:31:49.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:31:49.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:31:49.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-115964116995 2024-09-13T15:31:49.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:31:49.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964116995 2024-09-13T15:31:49.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 115964116995' 2024-09-13T15:31:49.259 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 115964116995 2024-09-13T15:31:49.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:31:49.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964116995 -lt 115964116995 2024-09-13T15:31:49.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:31:49.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:31:49.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509469 2024-09-13T15:31:49.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:31:49.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509469 2024-09-13T15:31:49.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:31:49.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509469 2024-09-13T15:31:49.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-09-13T15:31:49.599 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509469 2024-09-13T15:31:49.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:31:49.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509469 -lt 64424509469 2024-09-13T15:31:49.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:31:49.946 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:31:49.946 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:31:50.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:31:50.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:31:50.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:31:50.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:31:50.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:31:50.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:31:50.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:31:50.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:31:50.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:31:50.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:31:50.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:31:50.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:31:51.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:31:51.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:31:51.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:31:51.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: repair 1.0 2024-09-13T15:31:51.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-13T15:31:51.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-13T15:31:51.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:31:51.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:31:51.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:31:51.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:31:51.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T15:30:23.270380+0000 2024-09-13T15:31:51.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-13T15:31:51.774 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-09-13T15:31:51.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-13T15:30:23.270380+0000 2024-09-13T15:31:51.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T15:31:51.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:30:23.270380+0000 2024-09-13T15:31:51.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:31:51.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:31:51.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:31:51.791 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:31:51.791 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:31:51.791 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:31:51.791 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:31:51.791 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:31:52.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:30:23.270380+0000 '>' 2024-09-13T15:30:23.270380+0000 2024-09-13T15:31:52.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:31:53.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:31:53.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:31:53.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:31:53.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:31:53.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:31:53.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:31:53.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:31:53.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:30:23.270380+0000 '>' 2024-09-13T15:30:23.270380+0000 2024-09-13T15:31:53.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:31:54.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:31:54.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:31:54.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:31:54.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:31:54.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:31:54.552 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:31:54.552 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:31:54.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:31:52.804692+0000 '>' 2024-09-13T15:30:23.270380+0000 2024-09-13T15:31:54.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:31:54.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T15:31:56.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: wait_for_clean 2024-09-13T15:31:56.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:31:56.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:31:56.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:31:56.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:31:56.875 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:31:56.875 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:31:56.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:31:56.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:31:56.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:31:57.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:31:57.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:31:57.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:31:57.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:31:57.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:31:57.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:31:57.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:31:57.368 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:31:57.368 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:31:57.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:31:57.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:31:57.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:31:57.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018885 2024-09-13T15:31:57.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018885 2024-09-13T15:31:57.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018885' 2024-09-13T15:31:57.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:31:57.501 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:31:57.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964116998 2024-09-13T15:31:57.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964116998 2024-09-13T15:31:57.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018885 1-115964116998' 2024-09-13T15:31:57.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:31:57.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:31:57.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509472 2024-09-13T15:31:57.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509472 2024-09-13T15:31:57.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018885 1-115964116998 2-64424509472' 2024-09-13T15:31:57.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:31:57.776 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018885 2024-09-13T15:31:57.776 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:31:57.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:31:57.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018885 2024-09-13T15:31:57.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:31:57.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018885 2024-09-13T15:31:57.779 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 128849018885 2024-09-13T15:31:57.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-09-13T15:31:57.780 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:31:58.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018884 -lt 128849018885 2024-09-13T15:31:58.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:31:59.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:31:59.084 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:31:59.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018885 -lt 128849018885 2024-09-13T15:31:59.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:31:59.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-115964116998 2024-09-13T15:31:59.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:31:59.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:31:59.415 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-115964116998 2024-09-13T15:31:59.415 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:31:59.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964116998 2024-09-13T15:31:59.417 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 115964116998 2024-09-13T15:31:59.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 115964116998' 2024-09-13T15:31:59.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:31:59.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964116998 -lt 115964116998 2024-09-13T15:31:59.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:31:59.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509472 2024-09-13T15:31:59.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:31:59.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:31:59.768 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509472 2024-09-13T15:31:59.768 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:31:59.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509472 2024-09-13T15:31:59.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-09-13T15:31:59.769 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509472 2024-09-13T15:31:59.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:32:00.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509472 -lt 64424509472 2024-09-13T15:32:00.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:32:00.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:32:00.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:32:00.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:32:00.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:32:00.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:32:00.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:32:00.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:32:00.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:32:00.567 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:32:00.567 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:32:00.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:32:00.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:32:00.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:32:00.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:32:01.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:32:01.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:32:01.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:32:01.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: ceph pg dump pgs 2024-09-13T15:32:01.601 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T15:32:01.601 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-09-13T15:31:54.942411+0000 21'30 31:136 [1,0,2] 1 [1,0,2] 1 21'30 2024-09-13T15:31:52.804692+0000 21'30 2024-09-13T15:31:52.804692+0000 0 1 periodic scrub scheduled @ 2024-09-14T15:31:52.804692+0000 17 0 2024-09-13T15:32:01.601 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T15:32:01.601 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T15:32:01.601 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:32:01.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: flush_pg_stats 2024-09-13T15:32:01.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:32:01.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:32:01.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:32:01.922 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:32:01.922 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:32:01.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:32:01.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:32:01.922 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:32:02.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018887 2024-09-13T15:32:02.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018887 2024-09-13T15:32:02.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018887' 2024-09-13T15:32:02.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:32:02.061 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:32:02.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=115964117000 2024-09-13T15:32:02.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 115964117000 2024-09-13T15:32:02.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018887 1-115964117000' 2024-09-13T15:32:02.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:32:02.206 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:32:02.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509474 2024-09-13T15:32:02.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509474 2024-09-13T15:32:02.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018887 1-115964117000 2-64424509474' 2024-09-13T15:32:02.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:32:02.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018887 2024-09-13T15:32:02.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:32:02.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:32:02.339 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018887 2024-09-13T15:32:02.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:32:02.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018887 2024-09-13T15:32:02.341 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 128849018887 2024-09-13T15:32:02.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018887' 2024-09-13T15:32:02.342 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:32:02.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018886 -lt 128849018887 2024-09-13T15:32:02.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:32:03.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:32:03.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:32:03.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018887 -lt 128849018887 2024-09-13T15:32:03.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:32:03.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-115964117000 2024-09-13T15:32:03.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:32:03.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:32:03.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-115964117000 2024-09-13T15:32:03.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:32:03.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=115964117000 2024-09-13T15:32:03.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 115964117000' 2024-09-13T15:32:03.988 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 115964117000 2024-09-13T15:32:03.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:32:04.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 115964117000 -lt 115964117000 2024-09-13T15:32:04.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:32:04.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509474 2024-09-13T15:32:04.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:32:04.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:32:04.334 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509474 2024-09-13T15:32:04.334 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:32:04.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509474 2024-09-13T15:32:04.336 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509474 2024-09-13T15:32:04.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-09-13T15:32:04.336 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:32:04.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509474 -lt 64424509474 2024-09-13T15:32:04.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: ceph pg 1.0 query 2024-09-13T15:32:04.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-09-13T15:32:04.784 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:32:04.785 INFO:tasks.workunit.client.0.smithi112.stdout: "num_bytes": 210, 2024-09-13T15:32:04.785 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects": 30, 2024-09-13T15:32:04.785 INFO:tasks.workunit.client.0.smithi112.stdout: "num_object_clones": 0, 2024-09-13T15:32:04.785 INFO:tasks.workunit.client.0.smithi112.stdout: "num_object_copies": 90, 2024-09-13T15:32:04.785 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_missing_on_primary": 0, 2024-09-13T15:32:04.786 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_missing": 0, 2024-09-13T15:32:04.786 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_degraded": 0, 2024-09-13T15:32:04.786 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_misplaced": 0, 2024-09-13T15:32:04.786 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_unfound": 0, 2024-09-13T15:32:04.786 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_dirty": 30, 2024-09-13T15:32:04.786 INFO:tasks.workunit.client.0.smithi112.stdout: "num_whiteouts": 0, 2024-09-13T15:32:04.787 INFO:tasks.workunit.client.0.smithi112.stdout: "num_read": 0, 2024-09-13T15:32:04.787 INFO:tasks.workunit.client.0.smithi112.stdout: "num_read_kb": 0, 2024-09-13T15:32:04.787 INFO:tasks.workunit.client.0.smithi112.stdout: "num_write": 30, 2024-09-13T15:32:04.787 INFO:tasks.workunit.client.0.smithi112.stdout: "num_write_kb": 30, 2024-09-13T15:32:04.787 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrub_errors": 0, 2024-09-13T15:32:04.787 INFO:tasks.workunit.client.0.smithi112.stdout: "num_shallow_scrub_errors": 0, 2024-09-13T15:32:04.787 INFO:tasks.workunit.client.0.smithi112.stdout: "num_deep_scrub_errors": 0, 2024-09-13T15:32:04.788 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_recovered": 26, 2024-09-13T15:32:04.788 INFO:tasks.workunit.client.0.smithi112.stdout: "num_bytes_recovered": 182, 2024-09-13T15:32:04.788 INFO:tasks.workunit.client.0.smithi112.stdout: "num_keys_recovered": 0, 2024-09-13T15:32:04.788 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_omap": 0, 2024-09-13T15:32:04.788 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_hit_set_archive": 0, 2024-09-13T15:32:04.788 INFO:tasks.workunit.client.0.smithi112.stdout: "num_bytes_hit_set_archive": 0, 2024-09-13T15:32:04.788 INFO:tasks.workunit.client.0.smithi112.stdout: "num_flush": 0, 2024-09-13T15:32:04.789 INFO:tasks.workunit.client.0.smithi112.stdout: "num_flush_kb": 0, 2024-09-13T15:32:04.789 INFO:tasks.workunit.client.0.smithi112.stdout: "num_evict": 0, 2024-09-13T15:32:04.789 INFO:tasks.workunit.client.0.smithi112.stdout: "num_evict_kb": 0, 2024-09-13T15:32:04.789 INFO:tasks.workunit.client.0.smithi112.stdout: "num_promote": 0, 2024-09-13T15:32:04.789 INFO:tasks.workunit.client.0.smithi112.stdout: "num_flush_mode_high": 0, 2024-09-13T15:32:04.789 INFO:tasks.workunit.client.0.smithi112.stdout: "num_flush_mode_low": 0, 2024-09-13T15:32:04.789 INFO:tasks.workunit.client.0.smithi112.stdout: "num_evict_mode_some": 0, 2024-09-13T15:32:04.790 INFO:tasks.workunit.client.0.smithi112.stdout: "num_evict_mode_full": 0, 2024-09-13T15:32:04.790 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_pinned": 0, 2024-09-13T15:32:04.790 INFO:tasks.workunit.client.0.smithi112.stdout: "num_legacy_snapsets": 0, 2024-09-13T15:32:04.790 INFO:tasks.workunit.client.0.smithi112.stdout: "num_large_omap_objects": 0, 2024-09-13T15:32:04.790 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_manifest": 0, 2024-09-13T15:32:04.790 INFO:tasks.workunit.client.0.smithi112.stdout: "num_omap_bytes": 0, 2024-09-13T15:32:04.791 INFO:tasks.workunit.client.0.smithi112.stdout: "num_omap_keys": 0, 2024-09-13T15:32:04.791 INFO:tasks.workunit.client.0.smithi112.stdout: "num_objects_repaired": 26 2024-09-13T15:32:04.791 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:32:04.792 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-09-13T15:32:04.792 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-09-13T15:32:04.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: COUNT=26 2024-09-13T15:32:04.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: test 26 = 26 2024-09-13T15:32:04.930 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: expr 3 - 1 2024-09-13T15:32:04.931 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: seq 0 2 2024-09-13T15:32:04.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:32:04.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-09-13T15:32:04.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-09-13T15:32:04.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-09-13T15:32:04.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-13T15:32:04.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-09-13T15:32:05.231 INFO:tasks.workunit.client.0.smithi112.stderr:dumped all 2024-09-13T15:32:05.248 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:32:05.248 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:32:05.248 INFO:tasks.workunit.client.0.smithi112.stdout: "up_from": 30, 2024-09-13T15:32:05.248 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 128849018887, 2024-09-13T15:32:05.248 INFO:tasks.workunit.client.0.smithi112.stdout: "num_pgs": 1, 2024-09-13T15:32:05.248 INFO:tasks.workunit.client.0.smithi112.stdout: "num_osds": 1, 2024-09-13T15:32:05.248 INFO:tasks.workunit.client.0.smithi112.stdout: "num_per_pool_osds": 1, 2024-09-13T15:32:05.248 INFO:tasks.workunit.client.0.smithi112.stdout: "num_per_pool_omap_osds": 1, 2024-09-13T15:32:05.248 INFO:tasks.workunit.client.0.smithi112.stdout: "kb": 104857600, 2024-09-13T15:32:05.248 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used": 28488, 2024-09-13T15:32:05.248 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_data": 376, 2024-09-13T15:32:05.248 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_omap": 4, 2024-09-13T15:32:05.249 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_meta": 28091, 2024-09-13T15:32:05.249 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_avail": 104829112, 2024-09-13T15:32:05.249 INFO:tasks.workunit.client.0.smithi112.stdout: "statfs": { 2024-09-13T15:32:05.249 INFO:tasks.workunit.client.0.smithi112.stdout: "total": 107374182400, 2024-09-13T15:32:05.249 INFO:tasks.workunit.client.0.smithi112.stdout: "available": 107345010688, 2024-09-13T15:32:05.249 INFO:tasks.workunit.client.0.smithi112.stdout: "internally_reserved": 0, 2024-09-13T15:32:05.249 INFO:tasks.workunit.client.0.smithi112.stdout: "allocated": 385024, 2024-09-13T15:32:05.249 INFO:tasks.workunit.client.0.smithi112.stdout: "data_stored": 206078, 2024-09-13T15:32:05.249 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed": 0, 2024-09-13T15:32:05.249 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed_allocated": 0, 2024-09-13T15:32:05.249 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed_original": 0, 2024-09-13T15:32:05.249 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_allocated": 4402, 2024-09-13T15:32:05.249 INFO:tasks.workunit.client.0.smithi112.stdout: "internal_metadata": 28765902 2024-09-13T15:32:05.250 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:32:05.250 INFO:tasks.workunit.client.0.smithi112.stdout: "hb_peers": [ 2024-09-13T15:32:05.250 INFO:tasks.workunit.client.0.smithi112.stdout: 1, 2024-09-13T15:32:05.250 INFO:tasks.workunit.client.0.smithi112.stdout: 2 2024-09-13T15:32:05.250 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:32:05.250 INFO:tasks.workunit.client.0.smithi112.stdout: "snap_trim_queue_len": 0, 2024-09-13T15:32:05.250 INFO:tasks.workunit.client.0.smithi112.stdout: "num_snap_trimming": 0, 2024-09-13T15:32:05.250 INFO:tasks.workunit.client.0.smithi112.stdout: "num_shards_repaired": 13, 2024-09-13T15:32:05.250 INFO:tasks.workunit.client.0.smithi112.stdout: "op_queue_age_hist": { 2024-09-13T15:32:05.250 INFO:tasks.workunit.client.0.smithi112.stdout: "histogram": [], 2024-09-13T15:32:05.250 INFO:tasks.workunit.client.0.smithi112.stdout: "upper_bound": 1 2024-09-13T15:32:05.250 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:32:05.250 INFO:tasks.workunit.client.0.smithi112.stdout: "perf_stat": { 2024-09-13T15:32:05.251 INFO:tasks.workunit.client.0.smithi112.stdout: "commit_latency_ms": 0, 2024-09-13T15:32:05.251 INFO:tasks.workunit.client.0.smithi112.stdout: "apply_latency_ms": 0, 2024-09-13T15:32:05.251 INFO:tasks.workunit.client.0.smithi112.stdout: "commit_latency_ns": 0, 2024-09-13T15:32:05.251 INFO:tasks.workunit.client.0.smithi112.stdout: "apply_latency_ns": 0 2024-09-13T15:32:05.251 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:32:05.251 INFO:tasks.workunit.client.0.smithi112.stdout: "alerts": [] 2024-09-13T15:32:05.251 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:32:05.251 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-13T15:32:05.252 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-09-13T15:32:05.554 INFO:tasks.workunit.client.0.smithi112.stderr:dumped all 2024-09-13T15:32:05.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=13 2024-09-13T15:32:05.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 13 = 13 2024-09-13T15:32:05.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:32:05.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 1 = 0 -o 1 = 1 ']' 2024-09-13T15:32:05.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-09-13T15:32:05.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-09-13T15:32:05.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-13T15:32:05.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-09-13T15:32:05.870 INFO:tasks.workunit.client.0.smithi112.stderr:dumped all 2024-09-13T15:32:05.885 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:32:05.886 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:32:05.886 INFO:tasks.workunit.client.0.smithi112.stdout: "up_from": 27, 2024-09-13T15:32:05.886 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 115964117001, 2024-09-13T15:32:05.886 INFO:tasks.workunit.client.0.smithi112.stdout: "num_pgs": 1, 2024-09-13T15:32:05.886 INFO:tasks.workunit.client.0.smithi112.stdout: "num_osds": 1, 2024-09-13T15:32:05.886 INFO:tasks.workunit.client.0.smithi112.stdout: "num_per_pool_osds": 1, 2024-09-13T15:32:05.886 INFO:tasks.workunit.client.0.smithi112.stdout: "num_per_pool_omap_osds": 1, 2024-09-13T15:32:05.886 INFO:tasks.workunit.client.0.smithi112.stdout: "kb": 104857600, 2024-09-13T15:32:05.886 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used": 28488, 2024-09-13T15:32:05.886 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_data": 376, 2024-09-13T15:32:05.886 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_omap": 4, 2024-09-13T15:32:05.886 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_meta": 28091, 2024-09-13T15:32:05.886 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_avail": 104829112, 2024-09-13T15:32:05.886 INFO:tasks.workunit.client.0.smithi112.stdout: "statfs": { 2024-09-13T15:32:05.887 INFO:tasks.workunit.client.0.smithi112.stdout: "total": 107374182400, 2024-09-13T15:32:05.887 INFO:tasks.workunit.client.0.smithi112.stdout: "available": 107345010688, 2024-09-13T15:32:05.887 INFO:tasks.workunit.client.0.smithi112.stdout: "internally_reserved": 0, 2024-09-13T15:32:05.887 INFO:tasks.workunit.client.0.smithi112.stdout: "allocated": 385024, 2024-09-13T15:32:05.887 INFO:tasks.workunit.client.0.smithi112.stdout: "data_stored": 206078, 2024-09-13T15:32:05.887 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed": 0, 2024-09-13T15:32:05.887 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed_allocated": 0, 2024-09-13T15:32:05.887 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed_original": 0, 2024-09-13T15:32:05.887 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_allocated": 4398, 2024-09-13T15:32:05.887 INFO:tasks.workunit.client.0.smithi112.stdout: "internal_metadata": 28765906 2024-09-13T15:32:05.887 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:32:05.887 INFO:tasks.workunit.client.0.smithi112.stdout: "hb_peers": [ 2024-09-13T15:32:05.887 INFO:tasks.workunit.client.0.smithi112.stdout: 0, 2024-09-13T15:32:05.888 INFO:tasks.workunit.client.0.smithi112.stdout: 2 2024-09-13T15:32:05.888 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:32:05.888 INFO:tasks.workunit.client.0.smithi112.stdout: "snap_trim_queue_len": 0, 2024-09-13T15:32:05.888 INFO:tasks.workunit.client.0.smithi112.stdout: "num_snap_trimming": 0, 2024-09-13T15:32:05.888 INFO:tasks.workunit.client.0.smithi112.stdout: "num_shards_repaired": 13, 2024-09-13T15:32:05.888 INFO:tasks.workunit.client.0.smithi112.stdout: "op_queue_age_hist": { 2024-09-13T15:32:05.888 INFO:tasks.workunit.client.0.smithi112.stdout: "histogram": [], 2024-09-13T15:32:05.888 INFO:tasks.workunit.client.0.smithi112.stdout: "upper_bound": 1 2024-09-13T15:32:05.888 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:32:05.888 INFO:tasks.workunit.client.0.smithi112.stdout: "perf_stat": { 2024-09-13T15:32:05.888 INFO:tasks.workunit.client.0.smithi112.stdout: "commit_latency_ms": 0, 2024-09-13T15:32:05.888 INFO:tasks.workunit.client.0.smithi112.stdout: "apply_latency_ms": 0, 2024-09-13T15:32:05.888 INFO:tasks.workunit.client.0.smithi112.stdout: "commit_latency_ns": 0, 2024-09-13T15:32:05.888 INFO:tasks.workunit.client.0.smithi112.stdout: "apply_latency_ns": 0 2024-09-13T15:32:05.888 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:32:05.889 INFO:tasks.workunit.client.0.smithi112.stdout: "alerts": [] 2024-09-13T15:32:05.889 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:32:05.889 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-13T15:32:05.889 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-09-13T15:32:06.168 INFO:tasks.workunit.client.0.smithi112.stderr:dumped all 2024-09-13T15:32:06.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=13 2024-09-13T15:32:06.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 13 = 13 2024-09-13T15:32:06.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:32:06.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 2 = 0 -o 2 = 1 ']' 2024-09-13T15:32:06.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:784: TEST_repair_stats_ec: repair=0 2024-09-13T15:32:06.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-13T15:32:06.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 )' 2024-09-13T15:32:06.473 INFO:tasks.workunit.client.0.smithi112.stderr:dumped all 2024-09-13T15:32:06.490 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:32:06.490 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 2, 2024-09-13T15:32:06.490 INFO:tasks.workunit.client.0.smithi112.stdout: "up_from": 15, 2024-09-13T15:32:06.490 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 64424509474, 2024-09-13T15:32:06.490 INFO:tasks.workunit.client.0.smithi112.stdout: "num_pgs": 1, 2024-09-13T15:32:06.490 INFO:tasks.workunit.client.0.smithi112.stdout: "num_osds": 1, 2024-09-13T15:32:06.490 INFO:tasks.workunit.client.0.smithi112.stdout: "num_per_pool_osds": 1, 2024-09-13T15:32:06.490 INFO:tasks.workunit.client.0.smithi112.stdout: "num_per_pool_omap_osds": 1, 2024-09-13T15:32:06.490 INFO:tasks.workunit.client.0.smithi112.stdout: "kb": 104857600, 2024-09-13T15:32:06.490 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used": 27216, 2024-09-13T15:32:06.490 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_data": 376, 2024-09-13T15:32:06.490 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_omap": 1, 2024-09-13T15:32:06.491 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_meta": 26814, 2024-09-13T15:32:06.491 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_avail": 104830384, 2024-09-13T15:32:06.491 INFO:tasks.workunit.client.0.smithi112.stdout: "statfs": { 2024-09-13T15:32:06.491 INFO:tasks.workunit.client.0.smithi112.stdout: "total": 107374182400, 2024-09-13T15:32:06.491 INFO:tasks.workunit.client.0.smithi112.stdout: "available": 107346313216, 2024-09-13T15:32:06.491 INFO:tasks.workunit.client.0.smithi112.stdout: "internally_reserved": 0, 2024-09-13T15:32:06.491 INFO:tasks.workunit.client.0.smithi112.stdout: "allocated": 385024, 2024-09-13T15:32:06.491 INFO:tasks.workunit.client.0.smithi112.stdout: "data_stored": 206078, 2024-09-13T15:32:06.491 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed": 0, 2024-09-13T15:32:06.491 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed_allocated": 0, 2024-09-13T15:32:06.491 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed_original": 0, 2024-09-13T15:32:06.491 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_allocated": 1595, 2024-09-13T15:32:06.491 INFO:tasks.workunit.client.0.smithi112.stdout: "internal_metadata": 27457989 2024-09-13T15:32:06.491 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:32:06.492 INFO:tasks.workunit.client.0.smithi112.stdout: "hb_peers": [ 2024-09-13T15:32:06.492 INFO:tasks.workunit.client.0.smithi112.stdout: 0, 2024-09-13T15:32:06.492 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:32:06.492 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:32:06.492 INFO:tasks.workunit.client.0.smithi112.stdout: "snap_trim_queue_len": 0, 2024-09-13T15:32:06.492 INFO:tasks.workunit.client.0.smithi112.stdout: "num_snap_trimming": 0, 2024-09-13T15:32:06.492 INFO:tasks.workunit.client.0.smithi112.stdout: "num_shards_repaired": 0, 2024-09-13T15:32:06.492 INFO:tasks.workunit.client.0.smithi112.stdout: "op_queue_age_hist": { 2024-09-13T15:32:06.492 INFO:tasks.workunit.client.0.smithi112.stdout: "histogram": [], 2024-09-13T15:32:06.492 INFO:tasks.workunit.client.0.smithi112.stdout: "upper_bound": 1 2024-09-13T15:32:06.492 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:32:06.492 INFO:tasks.workunit.client.0.smithi112.stdout: "perf_stat": { 2024-09-13T15:32:06.492 INFO:tasks.workunit.client.0.smithi112.stdout: "commit_latency_ms": 0, 2024-09-13T15:32:06.492 INFO:tasks.workunit.client.0.smithi112.stdout: "apply_latency_ms": 0, 2024-09-13T15:32:06.492 INFO:tasks.workunit.client.0.smithi112.stdout: "commit_latency_ns": 0, 2024-09-13T15:32:06.493 INFO:tasks.workunit.client.0.smithi112.stdout: "apply_latency_ns": 0 2024-09-13T15:32:06.493 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:32:06.493 INFO:tasks.workunit.client.0.smithi112.stdout: "alerts": [] 2024-09-13T15:32:06.493 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:32:06.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-13T15:32:06.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 ).num_shards_repaired' 2024-09-13T15:32:06.766 INFO:tasks.workunit.client.0.smithi112.stderr:dumped all 2024-09-13T15:32:06.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=0 2024-09-13T15:32:06.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 0 = 0 2024-09-13T15:32:06.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:792: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-13T15:32:06.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:792: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum 2024-09-13T15:32:07.092 INFO:tasks.workunit.client.0.smithi112.stderr:dumped all 2024-09-13T15:32:07.109 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:32:07.109 INFO:tasks.workunit.client.0.smithi112.stdout: "up_from": 0, 2024-09-13T15:32:07.109 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 0, 2024-09-13T15:32:07.109 INFO:tasks.workunit.client.0.smithi112.stdout: "num_pgs": 3, 2024-09-13T15:32:07.109 INFO:tasks.workunit.client.0.smithi112.stdout: "num_osds": 3, 2024-09-13T15:32:07.109 INFO:tasks.workunit.client.0.smithi112.stdout: "num_per_pool_osds": 3, 2024-09-13T15:32:07.109 INFO:tasks.workunit.client.0.smithi112.stdout: "num_per_pool_omap_osds": 3, 2024-09-13T15:32:07.109 INFO:tasks.workunit.client.0.smithi112.stdout: "kb": 314572800, 2024-09-13T15:32:07.110 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used": 84192, 2024-09-13T15:32:07.110 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_data": 1128, 2024-09-13T15:32:07.110 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_omap": 10, 2024-09-13T15:32:07.110 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_used_meta": 82997, 2024-09-13T15:32:07.110 INFO:tasks.workunit.client.0.smithi112.stdout: "kb_avail": 314488608, 2024-09-13T15:32:07.110 INFO:tasks.workunit.client.0.smithi112.stdout: "statfs": { 2024-09-13T15:32:07.110 INFO:tasks.workunit.client.0.smithi112.stdout: "total": 322122547200, 2024-09-13T15:32:07.110 INFO:tasks.workunit.client.0.smithi112.stdout: "available": 322036334592, 2024-09-13T15:32:07.110 INFO:tasks.workunit.client.0.smithi112.stdout: "internally_reserved": 0, 2024-09-13T15:32:07.110 INFO:tasks.workunit.client.0.smithi112.stdout: "allocated": 1155072, 2024-09-13T15:32:07.110 INFO:tasks.workunit.client.0.smithi112.stdout: "data_stored": 618234, 2024-09-13T15:32:07.110 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed": 0, 2024-09-13T15:32:07.110 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed_allocated": 0, 2024-09-13T15:32:07.111 INFO:tasks.workunit.client.0.smithi112.stdout: "data_compressed_original": 0, 2024-09-13T15:32:07.111 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_allocated": 10395, 2024-09-13T15:32:07.111 INFO:tasks.workunit.client.0.smithi112.stdout: "internal_metadata": 84989797 2024-09-13T15:32:07.111 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:32:07.111 INFO:tasks.workunit.client.0.smithi112.stdout: "hb_peers": [], 2024-09-13T15:32:07.111 INFO:tasks.workunit.client.0.smithi112.stdout: "snap_trim_queue_len": 0, 2024-09-13T15:32:07.111 INFO:tasks.workunit.client.0.smithi112.stdout: "num_snap_trimming": 0, 2024-09-13T15:32:07.111 INFO:tasks.workunit.client.0.smithi112.stdout: "num_shards_repaired": 26, 2024-09-13T15:32:07.111 INFO:tasks.workunit.client.0.smithi112.stdout: "op_queue_age_hist": { 2024-09-13T15:32:07.111 INFO:tasks.workunit.client.0.smithi112.stdout: "histogram": [], 2024-09-13T15:32:07.111 INFO:tasks.workunit.client.0.smithi112.stdout: "upper_bound": 1 2024-09-13T15:32:07.111 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:32:07.111 INFO:tasks.workunit.client.0.smithi112.stdout: "perf_stat": { 2024-09-13T15:32:07.111 INFO:tasks.workunit.client.0.smithi112.stdout: "commit_latency_ms": 0, 2024-09-13T15:32:07.112 INFO:tasks.workunit.client.0.smithi112.stdout: "apply_latency_ms": 0, 2024-09-13T15:32:07.112 INFO:tasks.workunit.client.0.smithi112.stdout: "commit_latency_ns": 0, 2024-09-13T15:32:07.112 INFO:tasks.workunit.client.0.smithi112.stdout: "apply_latency_ns": 0 2024-09-13T15:32:07.112 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:32:07.112 INFO:tasks.workunit.client.0.smithi112.stdout: "alerts": [], 2024-09-13T15:32:07.112 INFO:tasks.workunit.client.0.smithi112.stdout: "network_ping_times": [] 2024-09-13T15:32:07.112 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:32:07.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-13T15:32:07.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-09-13T15:32:07.426 INFO:tasks.workunit.client.0.smithi112.stderr:dumped all 2024-09-13T15:32:07.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: COUNT=26 2024-09-13T15:32:07.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:794: TEST_repair_stats_ec: test 26 = 26 2024-09-13T15:32:07.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T15:32:07.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:32:07.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:32:07.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:32:07.445 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:32:07.445 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:32:07.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:32:07.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:32:07.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:32:07.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:32:07.758 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:32:07.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:32:07.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:32:07.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:32:07.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:32:07.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:32:07.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:32:07.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:32:07.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:32:07.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:32:07.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:32:07.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:32:07.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:32:07.793 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:32:07.793 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:32:07.793 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:32:07.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:32:07.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:32:07.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:32:07.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T15:32:07.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T15:32:07.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T15:32:07.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T15:32:07.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:32:07.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:32:07.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:32:07.796 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:32:07.796 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:32:07.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:32:07.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:32:07.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:32:07.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:32:07.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:32:07.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:32:07.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:32:07.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:32:07.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:32:07.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:32:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:32:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:32:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:32:07.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:32:07.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:32:07.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:32:07.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:32:07.809 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:32:07.809 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:32:07.809 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:32:07.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:32:07.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:32:07.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:32:07.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T15:32:07.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:32:07.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:32:07.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:32:07.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T15:32:07.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:32:07.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T15:32:07.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:32:07.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T15:32:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_request_scrub_priority td/osd-scrub-repair 2024-09-13T15:32:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6200: TEST_request_scrub_priority: local dir=td/osd-scrub-repair 2024-09-13T15:32:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6201: TEST_request_scrub_priority: local poolname=psr_pool 2024-09-13T15:32:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6202: TEST_request_scrub_priority: local objname=POBJ 2024-09-13T15:32:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6203: TEST_request_scrub_priority: local OBJECTS=64 2024-09-13T15:32:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6204: TEST_request_scrub_priority: local PGS=8 2024-09-13T15:32:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6206: TEST_request_scrub_priority: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-13T15:32:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T15:32:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:32:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:32:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:32:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T15:32:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-13T15:32:08.343 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:32:08.343 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:32:08.343 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:32:08.343 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:32:08.344 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:32:08.344 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:32:08.344 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:32:08.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-13T15:32:08.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:32:08.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:32:08.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:32:08.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:32:08.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:32:08.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:32:08.574 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:32:08.574 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:32:08.574 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:32:08.575 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:32:08.575 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:32:08.575 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:32:08.575 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:32:08.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:32:08.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T15:32:08.693 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:32:08.693 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:32:08.693 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:32:08.693 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:32:08.694 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:32:08.694 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:32:08.694 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:32:08.694 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:32:08.694 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:32:08.694 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:32:08.694 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:32:08.694 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:32:08.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:32:08.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T15:32:08.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6207: TEST_request_scrub_priority: run_mgr td/osd-scrub-repair x 2024-09-13T15:32:08.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T15:32:08.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:32:08.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:32:08.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:32:08.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T15:32:08.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:32:10.488 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:32:10.488 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:32:10.488 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:32:10.489 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:32:10.489 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:32:10.489 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:32:10.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:32:10.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:32:10.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:32:10.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6208: TEST_request_scrub_priority: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-09-13T15:32:10.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6209: TEST_request_scrub_priority: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-09-13T15:32:10.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6210: TEST_request_scrub_priority: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-09-13T15:32:10.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:32:10.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:32:10.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:32:10.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:32:10.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:32:10.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:32:10.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:32:10.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:32:10.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:32:10.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:32:10.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:32:10.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:32:10.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:32:10.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:32:10.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:32:10.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:32:10.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:32:10.532 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:32:10.532 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:32:10.532 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:32:10.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:32:10.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:32:10.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:32:10.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:32:10.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:32:10.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:32:10.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:32:10.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:32:10.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:32:10.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:32:10.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:32:10.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:32:10.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-09-13T15:32:10.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:32:10.533 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:32:10.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d012aeaa-04ef-4039-95f7-036c71037fcc 2024-09-13T15:32:10.534 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 d012aeaa-04ef-4039-95f7-036c71037fcc 2024-09-13T15:32:10.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d012aeaa-04ef-4039-95f7-036c71037fcc' 2024-09-13T15:32:10.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:32:10.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD6WuRmY2LmIBAAnvwXMD319qGMmYXWkYLCug== 2024-09-13T15:32:10.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD6WuRmY2LmIBAAnvwXMD319qGMmYXWkYLCug=="}' 2024-09-13T15:32:10.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d012aeaa-04ef-4039-95f7-036c71037fcc -i td/osd-scrub-repair/0/new.json 2024-09-13T15:32:10.857 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:32:10.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T15:32:10.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQD6WuRmY2LmIBAAnvwXMD319qGMmYXWkYLCug== --osd-uuid d012aeaa-04ef-4039-95f7-036c71037fcc 2024-09-13T15:32:10.905 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:32:10.901+0000 7f67c6e647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:32:10.941 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:32:10.937+0000 7f67c6e647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:32:10.942 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:32:10.941+0000 7f67c6e647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:32:10.942 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:32:10.941+0000 7f67c6e647c0 -1 bdev(0x561126032800 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:32:10.943 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:32:10.941+0000 7f67c6e647c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T15:32:19.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T15:32:19.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:32:19.038 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:32:19.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:32:19.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:32:19.451 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:32:19.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:32:19.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:32:19.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:32:19.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-09-13T15:32:19.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:32:19.488 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:32:19.484+0000 7f28844737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:32:19.556 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:32:19.552+0000 7f28844737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:32:19.557 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:32:19.552+0000 7f28844737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:32:19.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:32:19.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:32:19.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:32:19.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:32:19.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:32:19.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:19.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:32:19.780 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:32:19.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:19.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:20.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:20.841 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:32:20.836+0000 7f28844737c0 -1 Falling back to public interface 2024-09-13T15:32:21.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:21.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:21.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:32:21.138 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:32:21.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:21.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:21.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:22.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:22.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:22.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:32:22.474 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:32:22.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:22.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:22.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:23.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:23.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:23.799 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:32:23.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:32:23.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:23.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:23.874 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:32:23.868+0000 7f28844737c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:32:24.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:25.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:25.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:25.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:32:25.128 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:32:25.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:25.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:25.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:26.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:26.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:26.466 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:32:26.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:32:26.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:26.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:26.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:27.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:27.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:27.957 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:32:27.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:32:27.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:27.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:28.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:29.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:29.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:29.281 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:32:29.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:32:29.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:29.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:29.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:30.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:30.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:30.653 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:32:30.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:32:30.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:30.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:30.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:31.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:31.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:31.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:32:31.992 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:32:31.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:31.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:32.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:33.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:33.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:33.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:32:33.314 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:32:33.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:33.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:33.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:34.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:34.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:34.701 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:32:34.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:32:34.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:34.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:35.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:36.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:36.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:36.171 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:32:36.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:32:36.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:36.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:36.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:37.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:37.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:37.502 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:32:37.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:32:37.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:37.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:37.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:38.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:38.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:38.923 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:32:38.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:32:38.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:38.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:39.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:40.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:40.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:40.257 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:32:40.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:32:40.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:40.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:40.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:41.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:41.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:41.584 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:32:41.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:32:41.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:41.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:41.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:42.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:42.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:42.995 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:32:42.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:32:42.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:42.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:43.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:44.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:44.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:44.334 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:32:44.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:32:44.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:44.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:44.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:45.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:45.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:45.672 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:32:45.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:32:45.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:45.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:46.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:47.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:47.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:47.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:32:47.327 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:32:47.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:47.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:47.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:48.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:48.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:48.752 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:32:48.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:32:48.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:48.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:49.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:50.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:50.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:50.265 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:32:50.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:32:50.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:50.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:50.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:51.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:51.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:51.869 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:32:51.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:32:51.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:51.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:52.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:53.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:53.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:53.820 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:32:53.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:32:53.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:53.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:54.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:55.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:55.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:55.775 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:32:55.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:32:55.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:55.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:56.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:32:57.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:32:57.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:32:57.161 INFO:tasks.workunit.client.0.smithi112.stdout:26 2024-09-13T15:32:57.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 26 2024-09-13T15:32:57.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:32:57.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:32:57.493 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1587695371,v1:127.0.0.1:6803/1587695371] [v2:127.0.0.1:6804/1587695371,v1:127.0.0.1:6805/1587695371] exists,up d012aeaa-04ef-4039-95f7-036c71037fcc 2024-09-13T15:32:57.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:32:57.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:32:57.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:32:57.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6212: TEST_request_scrub_priority: create_pool psr_pool 8 8 2024-09-13T15:32:57.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool 8 8 2024-09-13T15:32:58.240 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'psr_pool' created 2024-09-13T15:32:58.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:32:59.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6213: TEST_request_scrub_priority: wait_for_clean 2024-09-13T15:32:59.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:32:59.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:32:59.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:32:59.265 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:32:59.266 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:32:59.266 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:32:59.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:32:59.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:32:59.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:32:59.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:32:59.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:32:59.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:32:59.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:32:59.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:32:59.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:32:59.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T15:32:59.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:32:59.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:32:59.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:33:00.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836488 2024-09-13T15:33:00.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836488 2024-09-13T15:33:00.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488' 2024-09-13T15:33:00.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:33:00.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836488 2024-09-13T15:33:00.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:33:00.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:33:00.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836488 2024-09-13T15:33:00.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:33:00.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836488 2024-09-13T15:33:00.022 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836488 2024-09-13T15:33:00.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-13T15:33:00.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:33:00.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836487 -lt 21474836488 2024-09-13T15:33:00.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:33:01.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:33:01.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:33:01.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836488 2024-09-13T15:33:01.717 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:33:01.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:33:01.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:33:02.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 8 == 0 2024-09-13T15:33:02.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:33:02.086 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:33:02.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:33:02.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:33:02.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:33:02.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:33:02.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:33:02.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=8 2024-09-13T15:33:02.390 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:33:02.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:33:02.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:33:02.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 = 8 2024-09-13T15:33:02.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:33:02.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:33:02.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6215: TEST_request_scrub_priority: local osd=0 2024-09-13T15:33:02.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: add_something td/osd-scrub-repair psr_pool POBJ noscrub 2024-09-13T15:33:02.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:33:02.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-09-13T15:33:02.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-09-13T15:33:02.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:33:02.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:33:02.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:33:03.216 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:33:03.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:33:03.734 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:33:03.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:33:03.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:33:03.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-09-13T15:33:04.058 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6217: TEST_request_scrub_priority: get_primary psr_pool POBJ 2024-09-13T15:33:04.058 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=psr_pool 2024-09-13T15:33:04.059 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=POBJ 2024-09-13T15:33:04.059 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:33:04.059 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map psr_pool POBJ 2024-09-13T15:33:04.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6217: TEST_request_scrub_priority: local primary=0 2024-09-13T15:33:04.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6218: TEST_request_scrub_priority: get_pg psr_pool POBJ 2024-09-13T15:33:04.388 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=psr_pool 2024-09-13T15:33:04.388 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=POBJ 2024-09-13T15:33:04.388 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T15:33:04.388 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map psr_pool POBJ 2024-09-13T15:33:04.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6218: TEST_request_scrub_priority: local pg=1.1 2024-09-13T15:33:04.711 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: ceph osd dump 2024-09-13T15:33:04.712 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: awk '{ print $2 }' 2024-09-13T15:33:04.712 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: grep '^pool.*['\'']psr_pool['\'']' 2024-09-13T15:33:05.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: poolid=1 2024-09-13T15:33:05.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: local otherpgs 2024-09-13T15:33:05.022 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: expr 8 - 1 2024-09-13T15:33:05.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: seq 0 7 2024-09-13T15:33:05.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-13T15:33:05.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.0 2024-09-13T15:33:05.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.0 = 1.1 ']' 2024-09-13T15:33:05.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 ' 2024-09-13T15:33:05.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-13T15:33:05.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T15:33:05.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:05.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:05.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T15:33:05.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:05.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.0 schedule-scrub 1.0 2024-09-13T15:33:05.504 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:33:05.504 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T15:33:05.504 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:33:05.504 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-06T15:31:25.506289+0000" 2024-09-13T15:33:05.504 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:33:05.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-13T15:33:05.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.1 2024-09-13T15:33:05.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.1 = 1.1 ']' 2024-09-13T15:33:05.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: continue 2024-09-13T15:33:05.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-13T15:33:05.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.2 2024-09-13T15:33:05.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.2 = 1.1 ']' 2024-09-13T15:33:05.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 ' 2024-09-13T15:33:05.518 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-13T15:33:05.518 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T15:33:05.518 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:05.518 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:05.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T15:33:05.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:05.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.2 schedule-scrub 1.2 2024-09-13T15:33:05.957 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:33:05.957 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T15:33:05.957 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:33:05.957 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-06T15:31:25.959015+0000" 2024-09-13T15:33:05.957 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:33:05.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-13T15:33:05.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.3 2024-09-13T15:33:05.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.3 = 1.1 ']' 2024-09-13T15:33:05.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 ' 2024-09-13T15:33:05.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-13T15:33:05.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T15:33:05.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:05.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:05.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T15:33:06.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:06.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.3 schedule-scrub 1.3 2024-09-13T15:33:06.410 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:33:06.410 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T15:33:06.410 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:33:06.410 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-06T15:31:26.412318+0000" 2024-09-13T15:33:06.410 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:33:06.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-13T15:33:06.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.4 2024-09-13T15:33:06.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.4 = 1.1 ']' 2024-09-13T15:33:06.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 ' 2024-09-13T15:33:06.424 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-13T15:33:06.424 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T15:33:06.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:06.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:06.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T15:33:06.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:06.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.4 schedule-scrub 1.4 2024-09-13T15:33:06.872 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:33:06.872 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T15:33:06.872 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:33:06.873 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-06T15:31:26.874617+0000" 2024-09-13T15:33:06.873 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:33:06.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-13T15:33:06.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.5 2024-09-13T15:33:06.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.5 = 1.1 ']' 2024-09-13T15:33:06.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 ' 2024-09-13T15:33:06.888 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-13T15:33:06.888 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T15:33:06.888 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:06.888 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:06.888 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T15:33:07.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:07.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.5 schedule-scrub 1.5 2024-09-13T15:33:07.307 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:33:07.307 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T15:33:07.307 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:33:07.307 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-06T15:31:27.309511+0000" 2024-09-13T15:33:07.307 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:33:07.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-13T15:33:07.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.6 2024-09-13T15:33:07.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.6 = 1.1 ']' 2024-09-13T15:33:07.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 ' 2024-09-13T15:33:07.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-13T15:33:07.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T15:33:07.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:07.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:07.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T15:33:07.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:07.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.6 schedule-scrub 1.6 2024-09-13T15:33:07.760 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:33:07.760 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T15:33:07.760 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:33:07.760 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-06T15:31:27.762139+0000" 2024-09-13T15:33:07.760 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:33:07.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-09-13T15:33:07.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6224: TEST_request_scrub_priority: opg=1.7 2024-09-13T15:33:07.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: '[' 1.7 = 1.1 ']' 2024-09-13T15:33:07.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 1.7 ' 2024-09-13T15:33:07.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-13T15:33:07.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T15:33:07.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:07.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:07.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T15:33:08.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6229: TEST_request_scrub_priority: local other_last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:08.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: ceph tell 1.7 schedule-scrub 1.7 2024-09-13T15:33:08.204 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:33:08.204 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T15:33:08.204 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:33:08.204 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-06T15:31:28.205942+0000" 2024-09-13T15:33:08.204 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:33:08.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6234: TEST_request_scrub_priority: sleep 15 2024-09-13T15:33:23.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6235: TEST_request_scrub_priority: flush_pg_stats 2024-09-13T15:33:23.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:33:23.220 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:33:23.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T15:33:23.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:33:23.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:33:23.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:33:23.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836494 2024-09-13T15:33:23.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836494 2024-09-13T15:33:23.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836494' 2024-09-13T15:33:23.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:33:23.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836494 2024-09-13T15:33:23.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:33:23.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:33:23.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836494 2024-09-13T15:33:23.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:33:23.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836494 2024-09-13T15:33:23.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-13T15:33:23.763 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836494 2024-09-13T15:33:23.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:33:24.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836493 -lt 21474836494 2024-09-13T15:33:24.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:33:25.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:33:25.083 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:33:25.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836493 -lt 21474836494 2024-09-13T15:33:25.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:33:26.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T15:33:26.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:33:26.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836494 2024-09-13T15:33:26.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-09-13T15:33:26.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T15:33:26.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:26.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:26.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T15:33:27.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: local last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:27.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6239: TEST_request_scrub_priority: ceph tell 1.1 scrub 2024-09-13T15:33:27.204 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:33:27.204 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T15:33:27.204 INFO:tasks.workunit.client.0.smithi112.stdout: "must": true, 2024-09-13T15:33:27.204 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "0.000000" 2024-09-13T15:33:27.204 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:33:27.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6241: TEST_request_scrub_priority: ceph osd unset noscrub 2024-09-13T15:33:27.823 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is unset 2024-09-13T15:33:27.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6242: TEST_request_scrub_priority: ceph osd unset nodeep-scrub 2024-09-13T15:33:28.475 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is unset 2024-09-13T15:33:28.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6244: TEST_request_scrub_priority: wait_for_scrub 1.1 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:28.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.1 2024-09-13T15:33:28.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:28.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:33:28.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:33:28.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:28.499 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-13T15:33:28.500 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T15:33:28.500 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:28.500 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:28.500 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T15:33:28.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:32:57.957382+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:28.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:33:29.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:33:29.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:29.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-13T15:33:29.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T15:33:29.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:29.828 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:29.828 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T15:33:30.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:32:57.957382+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:30.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:33:31.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:33:31.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:31.188 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-13T15:33:31.188 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T15:33:31.188 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:31.189 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:31.189 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T15:33:31.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:32:57.957382+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:31.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:33:32.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:33:32.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:32.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-13T15:33:32.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T15:33:32.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:32.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:32.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T15:33:32.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:32:57.957382+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:32.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:33:33.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:33:33.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:33.864 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-13T15:33:33.864 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T15:33:33.864 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:33.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:33.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T15:33:34.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:33:28.000749+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:34.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:33:34.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-13T15:33:34.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.0 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:34.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T15:33:34.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:34.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:33:34.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:33:34.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:34.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:33:34.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:33:34.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:34.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:34.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:33:34.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T15:31:25.506289+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:34.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:33:35.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:33:35.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:35.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:33:35.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:33:35.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:35.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:35.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:33:35.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T15:31:25.506289+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:35.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:33:36.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:33:36.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:36.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:33:36.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:33:36.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:36.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:36.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:33:37.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T15:31:25.506289+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:37.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:33:38.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:33:38.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:38.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:33:38.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:33:38.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:38.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:38.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:33:38.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:33:32.010275+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:38.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:33:38.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-13T15:33:38.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.2 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:38.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.2 2024-09-13T15:33:38.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:38.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:33:38.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:33:38.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:38.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-09-13T15:33:38.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.2 2024-09-13T15:33:38.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:38.515 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:38.515 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-09-13T15:33:38.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:33:29.006048+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:38.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:33:38.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-13T15:33:38.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.3 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:38.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.3 2024-09-13T15:33:38.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:38.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:33:38.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:33:38.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:38.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-13T15:33:38.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T15:33:38.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:38.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:38.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T15:33:39.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:33:30.048246+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:39.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:33:39.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-13T15:33:39.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.4 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:39.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.4 2024-09-13T15:33:39.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:39.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:33:39.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:33:39.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:39.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-09-13T15:33:39.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.4 2024-09-13T15:33:39.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:39.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:39.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-09-13T15:33:39.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:33:31.019797+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:39.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:33:39.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-13T15:33:39.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.5 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:39.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.5 2024-09-13T15:33:39.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:39.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:33:39.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:33:39.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:39.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-09-13T15:33:39.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.5 2024-09-13T15:33:39.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:39.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-09-13T15:33:39.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:39.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:33:33.006547+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:39.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:33:39.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-13T15:33:39.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.6 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:39.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.6 2024-09-13T15:33:39.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:39.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:33:39.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:33:39.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:39.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-09-13T15:33:39.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2024-09-13T15:33:39.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:39.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:39.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-09-13T15:33:40.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:33:34.006030+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:40.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:33:40.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-13T15:33:40.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.7 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:40.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.7 2024-09-13T15:33:40.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:40.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:33:40.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:33:40.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:40.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-09-13T15:33:40.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.7 2024-09-13T15:33:40.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:40.095 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:40.095 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-09-13T15:33:40.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:33:35.025604+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:40.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:33:40.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6246: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-09-13T15:33:40.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6248: TEST_request_scrub_priority: wait_for_scrub 1.1 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:40.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.1 2024-09-13T15:33:40.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:40.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:33:40.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:33:40.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:33:40.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-09-13T15:33:40.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2024-09-13T15:33:40.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:33:40.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:33:40.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-09-13T15:33:40.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:33:28.000749+0000 '>' 2024-09-13T15:32:57.957382+0000 2024-09-13T15:33:40.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:33:40.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: grep -v purged_snaps 2024-09-13T15:33:40.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: grep 'log_channel.*scrub ok' td/osd-scrub-repair/osd.0.log 2024-09-13T15:33:40.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: head -1 2024-09-13T15:33:40.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: sed 's/.*[[]DBG[]]//' 2024-09-13T15:33:40.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6252: TEST_request_scrub_priority: grep -q 1.1 2024-09-13T15:33:40.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T15:33:40.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:33:40.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:33:40.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:33:40.714 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:33:40.714 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:33:40.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:33:40.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:33:40.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:33:40.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:33:40.822 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:33:40.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:33:40.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:33:40.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:33:40.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:33:40.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:33:40.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:33:40.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:33:40.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:33:40.828 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:33:40.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:33:40.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:33:40.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:33:40.837 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:33:40.837 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:33:40.837 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:33:40.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:33:40.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:33:40.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:33:40.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T15:33:40.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T15:33:40.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T15:33:40.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T15:33:40.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:33:40.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:33:40.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:33:40.840 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:33:40.840 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:33:40.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:33:40.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:33:40.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:33:40.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:33:40.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:33:40.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:33:40.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:33:40.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:33:40.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:33:40.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:33:40.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:33:40.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:33:40.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:33:40.845 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:33:40.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:33:40.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:33:40.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:33:40.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:33:40.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:33:40.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:33:40.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:33:40.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:33:40.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:33:40.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T15:33:40.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:33:40.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:33:40.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:33:40.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T15:33:40.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:33:40.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T15:33:40.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:33:40.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T15:33:40.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_scrub_warning td/osd-scrub-repair 2024-09-13T15:33:40.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5845: TEST_scrub_warning: local dir=td/osd-scrub-repair 2024-09-13T15:33:40.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5846: TEST_scrub_warning: local poolname=psr_pool 2024-09-13T15:33:40.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5847: TEST_scrub_warning: local objname=POBJ 2024-09-13T15:33:40.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5848: TEST_scrub_warning: local scrubs=5 2024-09-13T15:33:40.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5849: TEST_scrub_warning: local deep_scrubs=5 2024-09-13T15:33:40.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5850: TEST_scrub_warning: local i1_day=86400 2024-09-13T15:33:40.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_scrub_warning: calc 86400 '*' 7 2024-09-13T15:33:40.855 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: calc: awk 'BEGIN{print 86400 * 7}' 2024-09-13T15:33:40.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_scrub_warning: local i7_days=604800 2024-09-13T15:33:40.856 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5852: TEST_scrub_warning: calc 86400 '*' 14 2024-09-13T15:33:40.856 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: calc: awk 'BEGIN{print 86400 * 14}' 2024-09-13T15:33:40.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5852: TEST_scrub_warning: local i14_days=1209600 2024-09-13T15:33:40.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5853: TEST_scrub_warning: local overdue=0.5 2024-09-13T15:33:40.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5854: TEST_scrub_warning: calc 604800 + 86400 + '(' 604800 '*' 0.5 ')' 2024-09-13T15:33:40.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: calc: awk 'BEGIN{print 604800 + 86400 + ( 604800 * 0.5 )}' 2024-09-13T15:33:40.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5854: TEST_scrub_warning: local conf_overdue_seconds=993600 2024-09-13T15:33:40.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5855: TEST_scrub_warning: calc 1209600 + 86400 + '(' 1209600 '*' 0.5 ')' 2024-09-13T15:33:40.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: calc: awk 'BEGIN{print 1209600 + 86400 + ( 1209600 * 0.5 )}' 2024-09-13T15:33:40.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5855: TEST_scrub_warning: local pool_overdue_seconds=1900800 2024-09-13T15:33:40.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5857: TEST_scrub_warning: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-13T15:33:40.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T15:33:40.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:33:40.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:33:40.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:33:40.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T15:33:40.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-13T15:33:41.366 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:33:41.367 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:33:41.367 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:33:41.367 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:33:41.367 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:33:41.367 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:33:41.368 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:33:41.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-13T15:33:41.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:33:41.621 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:33:41.621 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:33:41.621 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:33:41.621 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:33:41.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:33:41.622 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:33:41.622 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:33:41.622 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:33:41.622 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:33:41.623 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:33:41.623 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:33:41.623 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:33:41.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:33:41.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T15:33:41.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:33:41.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:33:41.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:33:41.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:33:41.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:33:41.741 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:33:41.741 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:33:41.741 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:33:41.741 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:33:41.742 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:33:41.742 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:33:41.742 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:33:41.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:33:41.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T15:33:41.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: run_mgr td/osd-scrub-repair x --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2024-09-13T15:33:41.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T15:33:41.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:33:41.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:33:41.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:33:41.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T15:33:41.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:33:43.561 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:33:43.562 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:33:43.562 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:33:43.562 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:33:43.562 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:33:43.562 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:33:43.563 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:33:43.563 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:33:43.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2024-09-13T15:33:43.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5859: TEST_scrub_warning: run_osd td/osd-scrub-repair 0 --osd_scrub_backoff_ratio=0 2024-09-13T15:33:43.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:33:43.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:33:43.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:33:43.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:33:43.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:33:43.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:33:43.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:33:43.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:33:43.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:33:43.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:33:43.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:33:43.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:33:43.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:33:43.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:33:43.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:33:43.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:33:43.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:33:43.604 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:33:43.604 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:33:43.604 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:33:43.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:33:43.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:33:43.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:33:43.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:33:43.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:33:43.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:33:43.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:33:43.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:33:43.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:33:43.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:33:43.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:33:43.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:33:43.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_backoff_ratio=0 2024-09-13T15:33:43.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:33:43.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:33:43.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dc0346ea-4ca6-4581-9827-27544c13d181 2024-09-13T15:33:43.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 dc0346ea-4ca6-4581-9827-27544c13d181' 2024-09-13T15:33:43.608 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 dc0346ea-4ca6-4581-9827-27544c13d181 2024-09-13T15:33:43.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:33:43.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBXW+RmxFJQJRAAkyuScmA69zTCfZIIJ4CTGg== 2024-09-13T15:33:43.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBXW+RmxFJQJRAAkyuScmA69zTCfZIIJ4CTGg=="}' 2024-09-13T15:33:43.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dc0346ea-4ca6-4581-9827-27544c13d181 -i td/osd-scrub-repair/0/new.json 2024-09-13T15:33:43.930 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:33:43.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T15:33:43.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 --mkfs --key AQBXW+RmxFJQJRAAkyuScmA69zTCfZIIJ4CTGg== --osd-uuid dc0346ea-4ca6-4581-9827-27544c13d181 2024-09-13T15:33:43.985 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:33:43.982+0000 7fbf1e8787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:33:44.015 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:33:44.014+0000 7fbf1e8787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:33:44.016 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:33:44.014+0000 7fbf1e8787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:33:44.017 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:33:44.014+0000 7fbf1e8787c0 -1 bdev(0x5577d8e0a800 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:33:44.017 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:33:44.014+0000 7fbf1e8787c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T15:33:52.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T15:33:52.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:33:52.155 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:33:52.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:33:52.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:33:52.557 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:33:52.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:33:52.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 2024-09-13T15:33:52.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:33:52.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:33:52.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:33:52.591 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:33:52.590+0000 7fa183e047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:33:52.638 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:33:52.638+0000 7fa183e047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:33:52.640 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:33:52.638+0000 7fa183e047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:33:52.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:33:52.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:33:52.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:33:52.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:33:52.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:33:52.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:33:52.879 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:33:52.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:33:52.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:33:52.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:33:53.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:33:53.973 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:33:53.970+0000 7fa183e047c0 -1 Falling back to public interface 2024-09-13T15:33:54.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:33:54.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:33:54.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:33:54.189 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:33:54.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:33:54.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:33:54.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:33:55.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:33:55.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:33:55.515 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:33:55.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:33:55.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:33:55.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:33:55.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:33:56.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:33:56.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:33:56.849 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:33:56.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:33:56.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:33:56.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:33:56.931 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:33:56.930+0000 7fa183e047c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:33:57.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:33:58.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:33:58.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:33:58.174 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:33:58.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:33:58.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:33:58.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:33:58.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:33:59.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:33:59.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:33:59.598 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:33:59.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:33:59.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:33:59.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:33:59.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:00.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:00.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:00.961 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:34:00.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:34:00.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:00.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:01.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:02.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:02.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:02.337 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:34:02.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:34:02.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:02.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:02.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:03.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:03.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:03.677 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:34:03.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:34:03.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:03.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:04.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:05.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:05.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:05.169 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:34:05.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:34:05.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:05.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:05.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:06.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:06.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:06.510 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:34:06.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:34:06.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:06.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:06.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:07.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:07.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:07.913 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:34:07.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:34:07.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:07.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:08.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:09.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:09.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:09.426 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:34:09.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:34:09.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:09.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:09.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:10.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:10.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:10.760 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:34:10.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:34:10.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:10.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:11.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:12.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:12.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:12.217 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:34:12.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:34:12.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:12.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:12.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:13.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:13.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:13.709 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:34:13.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:34:13.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:13.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:14.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:15.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:15.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:15.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:34:15.167 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:34:15.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:15.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:15.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:16.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:16.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:16.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:34:16.827 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:34:16.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:16.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:17.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:18.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:18.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:18.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:34:18.227 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:34:18.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:18.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:19.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:20.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:20.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:20.254 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:34:20.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:34:20.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:20.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:20.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:21.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:21.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:21.903 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:34:21.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:34:21.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:21.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:22.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:23.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:23.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:23.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:34:23.722 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:34:23.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:23.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:24.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:25.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:25.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:25.510 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:34:25.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:34:25.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:25.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:25.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:26.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:26.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:26.850 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:34:26.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:34:26.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:26.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:27.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:34:28.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:34:28.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:34:28.622 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:34:28.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:34:28.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:34:28.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:34:28.977 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1652070657,v1:127.0.0.1:6803/1652070657] [v2:127.0.0.1:6804/1652070657,v1:127.0.0.1:6805/1652070657] exists,up dc0346ea-4ca6-4581-9827-27544c13d181 2024-09-13T15:34:28.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:34:28.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:34:28.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:34:28.979 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: expr 5 + 5 2024-09-13T15:34:28.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: seq 1 10 2024-09-13T15:34:28.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-13T15:34:28.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-1 1 1 2024-09-13T15:34:28.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-1 1 1 2024-09-13T15:34:29.544 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'psr_pool-1' created 2024-09-13T15:34:29.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:34:30.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-13T15:34:30.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:34:30.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:34:30.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:34:30.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:34:30.567 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:34:30.567 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:34:30.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:34:30.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:34:30.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:34:30.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:34:30.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:34:30.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:34:30.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:34:30.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:34:30.706 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:34:31.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T15:34:31.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:34:31.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:34:31.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:34:31.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836490 2024-09-13T15:34:31.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836490 2024-09-13T15:34:31.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836490' 2024-09-13T15:34:31.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:34:31.188 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836490 2024-09-13T15:34:31.188 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:34:31.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:34:31.190 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836490 2024-09-13T15:34:31.190 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:34:31.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836490 2024-09-13T15:34:31.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-13T15:34:31.192 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836490 2024-09-13T15:34:31.192 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:34:31.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836490 2024-09-13T15:34:31.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:34:32.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:34:32.497 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:34:32.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836490 2024-09-13T15:34:32.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:34:32.834 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:34:32.834 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:34:33.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:34:33.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:34:33.276 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:34:33.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:34:33.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:34:33.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:34:33.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:34:33.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:34:33.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:34:33.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:34:33.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:34:33.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:34:33.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:34:33.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:34:33.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:34:33.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 1 = 1 ']' 2024-09-13T15:34:33.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5867: TEST_scrub_warning: ceph osd pool set psr_pool-1 scrub_max_interval 1209600 2024-09-13T15:34:34.495 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 1 scrub_max_interval to 1209600 2024-09-13T15:34:34.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:34:34.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 1 = 6 ']' 2024-09-13T15:34:34.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-13T15:34:34.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-2 1 1 2024-09-13T15:34:34.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-2 1 1 2024-09-13T15:34:35.089 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'psr_pool-2' created 2024-09-13T15:34:35.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:34:36.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-13T15:34:36.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:34:36.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:34:36.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:34:36.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:34:36.115 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:34:36.115 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:34:36.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:34:36.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:34:36.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:34:36.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:34:36.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:34:36.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:34:36.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:34:36.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:34:36.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:34:36.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T15:34:36.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:34:36.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:34:36.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:34:36.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836492 2024-09-13T15:34:36.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836492 2024-09-13T15:34:36.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836492' 2024-09-13T15:34:36.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:34:36.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836492 2024-09-13T15:34:36.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:34:36.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:34:36.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836492 2024-09-13T15:34:36.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:34:36.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836492 2024-09-13T15:34:36.853 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836492 2024-09-13T15:34:36.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-13T15:34:36.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:34:37.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836491 -lt 21474836492 2024-09-13T15:34:37.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:34:38.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:34:38.180 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:34:38.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-13T15:34:38.504 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:34:38.504 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:34:38.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:34:38.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 2 == 0 2024-09-13T15:34:38.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:34:38.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:34:38.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:34:38.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:34:38.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:34:38.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:34:38.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:34:39.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-13T15:34:39.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:34:39.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:34:39.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:34:39.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 2 2024-09-13T15:34:39.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:34:39.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:34:39.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 2 = 1 ']' 2024-09-13T15:34:39.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:34:39.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 2 = 6 ']' 2024-09-13T15:34:39.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-13T15:34:39.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-3 1 1 2024-09-13T15:34:39.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-3 1 1 2024-09-13T15:34:40.106 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'psr_pool-3' created 2024-09-13T15:34:40.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:34:41.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-13T15:34:41.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:34:41.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:34:41.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:34:41.131 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:34:41.132 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:34:41.132 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:34:41.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:34:41.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:34:41.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:34:41.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:34:41.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:34:41.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:34:41.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:34:41.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:34:41.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:34:41.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T15:34:41.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:34:41.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:34:41.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:34:41.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836494 2024-09-13T15:34:41.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836494 2024-09-13T15:34:41.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836494' 2024-09-13T15:34:41.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:34:41.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836494 2024-09-13T15:34:41.791 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:34:41.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:34:41.793 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836494 2024-09-13T15:34:41.793 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:34:41.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836494 2024-09-13T15:34:41.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-13T15:34:41.794 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836494 2024-09-13T15:34:41.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:34:42.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836493 -lt 21474836494 2024-09-13T15:34:42.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:34:43.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:34:43.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:34:43.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836494 2024-09-13T15:34:43.429 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:34:43.430 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:34:43.430 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:34:43.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 3 == 0 2024-09-13T15:34:43.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:34:43.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:34:43.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:34:43.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:34:43.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:34:43.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:34:43.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:34:44.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-13T15:34:44.183 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:34:44.183 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:34:44.183 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:34:44.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 3 2024-09-13T15:34:44.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:34:44.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:34:44.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 3 = 1 ']' 2024-09-13T15:34:44.561 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:34:44.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 3 = 6 ']' 2024-09-13T15:34:44.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-13T15:34:44.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-4 1 1 2024-09-13T15:34:44.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-4 1 1 2024-09-13T15:34:44.999 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'psr_pool-4' created 2024-09-13T15:34:45.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:34:46.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-13T15:34:46.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:34:46.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:34:46.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:34:46.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:34:46.026 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:34:46.026 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:34:46.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:34:46.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:34:46.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:34:46.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:34:46.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:34:46.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:34:46.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:34:46.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:34:46.163 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:34:46.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T15:34:46.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:34:46.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:34:46.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:34:46.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836496 2024-09-13T15:34:46.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836496 2024-09-13T15:34:46.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836496' 2024-09-13T15:34:46.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:34:46.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836496 2024-09-13T15:34:46.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:34:46.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:34:46.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836496 2024-09-13T15:34:46.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:34:46.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836496 2024-09-13T15:34:46.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-09-13T15:34:46.689 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836496 2024-09-13T15:34:46.689 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:34:47.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836495 -lt 21474836496 2024-09-13T15:34:47.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:34:48.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:34:48.065 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:34:48.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836495 -lt 21474836496 2024-09-13T15:34:48.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:34:49.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T15:34:49.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:34:49.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836496 -lt 21474836496 2024-09-13T15:34:49.712 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:34:49.712 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:34:49.712 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:34:50.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T15:34:50.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:34:50.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:34:50.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:34:50.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:34:50.167 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:34:50.167 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:34:50.167 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:34:50.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T15:34:50.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:34:50.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:34:50.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:34:50.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T15:34:50.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:34:50.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:34:50.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 4 = 1 ']' 2024-09-13T15:34:50.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:34:50.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 4 = 6 ']' 2024-09-13T15:34:50.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-13T15:34:50.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-5 1 1 2024-09-13T15:34:50.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-5 1 1 2024-09-13T15:34:51.384 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'psr_pool-5' created 2024-09-13T15:34:51.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:34:52.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-13T15:34:52.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:34:52.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:34:52.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:34:52.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:34:52.406 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:34:52.406 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:34:52.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:34:52.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:34:52.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:34:52.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:34:52.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:34:52.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:34:52.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:34:52.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:34:52.543 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:34:52.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T15:34:52.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:34:52.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:34:52.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:34:53.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836498 2024-09-13T15:34:53.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836498 2024-09-13T15:34:53.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836498' 2024-09-13T15:34:53.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:34:53.086 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836498 2024-09-13T15:34:53.086 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:34:53.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:34:53.088 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836498 2024-09-13T15:34:53.088 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:34:53.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836498 2024-09-13T15:34:53.090 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836498 2024-09-13T15:34:53.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-13T15:34:53.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:34:53.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836498 2024-09-13T15:34:53.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:34:54.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:34:54.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:34:54.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836498 -lt 21474836498 2024-09-13T15:34:54.746 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:34:54.747 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:34:54.747 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:34:55.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-13T15:34:55.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:34:55.172 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:34:55.172 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:34:55.172 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:34:55.172 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:34:55.172 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:34:55.172 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:34:55.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-13T15:34:55.492 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:34:55.492 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:34:55.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:34:55.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-13T15:34:55.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:34:55.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:34:55.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 5 = 1 ']' 2024-09-13T15:34:55.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:34:55.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 5 = 6 ']' 2024-09-13T15:34:55.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-13T15:34:55.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-6 1 1 2024-09-13T15:34:55.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-6 1 1 2024-09-13T15:34:56.376 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'psr_pool-6' created 2024-09-13T15:34:56.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:34:57.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-13T15:34:57.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:34:57.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:34:57.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:34:57.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:34:57.402 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:34:57.402 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:34:57.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:34:57.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:34:57.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:34:57.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:34:57.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:34:57.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:34:57.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:34:57.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:34:57.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:34:57.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T15:34:57.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:34:57.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:34:57.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:34:58.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836500 2024-09-13T15:34:58.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836500 2024-09-13T15:34:58.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500' 2024-09-13T15:34:58.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:34:58.044 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:34:58.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836500 2024-09-13T15:34:58.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:34:58.046 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836500 2024-09-13T15:34:58.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:34:58.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836500 2024-09-13T15:34:58.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-13T15:34:58.048 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836500 2024-09-13T15:34:58.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:34:58.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836500 2024-09-13T15:34:58.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:34:59.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:34:59.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:34:59.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-13T15:34:59.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:34:59.671 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:34:59.671 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:35:00.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-13T15:35:00.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:35:00.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:35:00.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:35:00.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:35:00.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:35:00.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:35:00.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:35:00.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-13T15:35:00.497 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:35:00.497 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:35:00.497 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:35:00.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-13T15:35:00.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:35:00.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:35:00.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 6 = 1 ']' 2024-09-13T15:35:00.890 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:35:00.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 6 = 6 ']' 2024-09-13T15:35:00.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5871: TEST_scrub_warning: ceph osd pool set psr_pool-6 deep_scrub_interval 1209600 2024-09-13T15:35:01.363 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 6 deep_scrub_interval to 1209600 2024-09-13T15:35:01.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-13T15:35:01.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-7 1 1 2024-09-13T15:35:01.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-7 1 1 2024-09-13T15:35:01.873 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'psr_pool-7' created 2024-09-13T15:35:01.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:35:02.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-13T15:35:02.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:35:02.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:35:02.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:35:02.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:35:02.897 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:35:02.897 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:35:02.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:35:02.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:35:02.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:35:03.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:35:03.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:35:03.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:35:03.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:35:03.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:35:03.039 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:35:03.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T15:35:03.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:35:03.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:35:03.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:35:03.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-13T15:35:03.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-13T15:35:03.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-13T15:35:03.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:35:03.503 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-13T15:35:03.503 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:35:03.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:35:03.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-13T15:35:03.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:35:03.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-13T15:35:03.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-13T15:35:03.506 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836502 2024-09-13T15:35:03.506 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:35:03.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836502 2024-09-13T15:35:03.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:35:04.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:35:04.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:35:05.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-13T15:35:05.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:35:05.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:35:05.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:35:05.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 7 == 0 2024-09-13T15:35:05.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:35:05.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:35:05.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:35:05.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:35:05.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:35:05.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:35:05.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:35:05.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=7 2024-09-13T15:35:05.866 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:35:05.867 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:35:05.867 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:35:06.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 7 = 7 2024-09-13T15:35:06.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:35:06.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:35:06.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 7 = 1 ']' 2024-09-13T15:35:06.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:35:06.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 7 = 6 ']' 2024-09-13T15:35:06.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-13T15:35:06.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-8 1 1 2024-09-13T15:35:06.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-8 1 1 2024-09-13T15:35:06.832 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'psr_pool-8' created 2024-09-13T15:35:06.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:35:07.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-13T15:35:07.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:35:07.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:35:07.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:35:07.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:35:07.859 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:35:07.859 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:35:07.860 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:35:07.860 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:35:07.860 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:35:07.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:35:07.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:35:07.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:35:07.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:35:07.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:35:07.999 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:35:08.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T15:35:08.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:35:08.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:35:08.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:35:08.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-13T15:35:08.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-13T15:35:08.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-13T15:35:08.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:35:08.556 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-13T15:35:08.556 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:35:08.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:35:08.558 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-13T15:35:08.558 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:35:08.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-13T15:35:08.560 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836504 2024-09-13T15:35:08.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-13T15:35:08.560 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:35:08.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836504 2024-09-13T15:35:08.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:35:09.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:35:09.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:35:10.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836504 2024-09-13T15:35:10.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:35:11.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T15:35:11.207 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:35:11.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836504 2024-09-13T15:35:11.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:35:11.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:35:11.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:35:11.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 8 == 0 2024-09-13T15:35:11.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:35:11.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:35:11.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:35:11.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:35:11.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:35:11.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:35:11.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:35:12.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=8 2024-09-13T15:35:12.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:35:12.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:35:12.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:35:12.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 = 8 2024-09-13T15:35:12.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:35:12.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:35:12.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 8 = 1 ']' 2024-09-13T15:35:12.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:35:12.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 8 = 6 ']' 2024-09-13T15:35:12.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-13T15:35:12.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-9 1 1 2024-09-13T15:35:12.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-9 1 1 2024-09-13T15:35:13.075 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'psr_pool-9' created 2024-09-13T15:35:13.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:35:14.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-13T15:35:14.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:35:14.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:35:14.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:35:14.100 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:35:14.101 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:35:14.101 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:35:14.104 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:35:14.104 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:35:14.104 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:35:14.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:35:14.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:35:14.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:35:14.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:35:14.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:35:14.241 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:35:14.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T15:35:14.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:35:14.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:35:14.631 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:35:14.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-13T15:35:14.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-13T15:35:14.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-13T15:35:14.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:35:14.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-13T15:35:14.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:35:14.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:35:14.819 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-13T15:35:14.820 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:35:14.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-13T15:35:14.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-13T15:35:14.821 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836507 2024-09-13T15:35:14.822 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:35:15.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836507 2024-09-13T15:35:15.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:35:16.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:35:16.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:35:16.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-13T15:35:16.482 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:35:16.482 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:35:16.482 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:35:16.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 9 == 0 2024-09-13T15:35:16.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:35:16.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:35:16.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:35:16.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:35:16.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:35:16.886 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:35:16.886 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:35:17.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=9 2024-09-13T15:35:17.211 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:35:17.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:35:17.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:35:17.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 = 9 2024-09-13T15:35:17.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:35:17.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:35:17.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 9 = 1 ']' 2024-09-13T15:35:17.587 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:35:17.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 9 = 6 ']' 2024-09-13T15:35:17.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-09-13T15:35:17.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5863: TEST_scrub_warning: create_pool psr_pool-10 1 1 2024-09-13T15:35:17.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-10 1 1 2024-09-13T15:35:18.008 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'psr_pool-10' created 2024-09-13T15:35:18.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:35:19.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: wait_for_clean 2024-09-13T15:35:19.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:35:19.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:35:19.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:35:19.029 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:35:19.030 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:35:19.030 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:35:19.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:35:19.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:35:19.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:35:19.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:35:19.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:35:19.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:35:19.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:35:19.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:35:19.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:35:19.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T15:35:19.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:35:19.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:35:19.503 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:35:19.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836508 2024-09-13T15:35:19.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836508 2024-09-13T15:35:19.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508' 2024-09-13T15:35:19.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:35:19.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836508 2024-09-13T15:35:19.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:35:19.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:35:19.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836508 2024-09-13T15:35:19.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:35:19.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836508 2024-09-13T15:35:19.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-13T15:35:19.683 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836508 2024-09-13T15:35:19.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:35:19.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836508 2024-09-13T15:35:19.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:35:21.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:35:21.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:35:21.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836508 2024-09-13T15:35:21.336 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:35:21.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:35:21.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:35:21.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 10 == 0 2024-09-13T15:35:21.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:35:21.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:35:21.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:35:21.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:35:21.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:35:21.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:35:21.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:35:22.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=10 2024-09-13T15:35:22.062 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:35:22.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:35:22.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:35:22.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 = 10 2024-09-13T15:35:22.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:35:22.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:35:22.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5865: TEST_scrub_warning: '[' 10 = 1 ']' 2024-09-13T15:35:22.494 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:35:22.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5869: TEST_scrub_warning: '[' 10 = 6 ']' 2024-09-13T15:35:22.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5876: TEST_scrub_warning: local primary=0 2024-09-13T15:35:22.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: ceph osd set noscrub 2024-09-13T15:35:23.101 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:35:23.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: ceph osd set nodeep-scrub 2024-09-13T15:35:23.628 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:35:23.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5880: TEST_scrub_warning: ceph config set global osd_scrub_interval_randomize_ratio 0 2024-09-13T15:35:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5881: TEST_scrub_warning: ceph config set global osd_deep_scrub_randomize_ratio 0 2024-09-13T15:35:24.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5882: TEST_scrub_warning: ceph config set global osd_scrub_max_interval 604800 2024-09-13T15:35:25.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: ceph config set global osd_deep_scrub_interval 604800 2024-09-13T15:35:25.449 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: seq 1 5 2024-09-13T15:35:25.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-13T15:35:25.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5888: TEST_scrub_warning: '[' 1 = 1 ']' 2024-09-13T15:35:25.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: TEST_scrub_warning: overdue_seconds=1900800 2024-09-13T15:35:25.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 1900800 + 100 2024-09-13T15:35:25.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph tell 1.0 schedule-scrub 1900900 2024-09-13T15:35:25.590 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:35:25.590 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T15:35:25.590 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:35:25.590 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-08-22T15:32:05.592292+0000" 2024-09-13T15:35:25.590 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:35:25.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-13T15:35:25.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5888: TEST_scrub_warning: '[' 2 = 1 ']' 2024-09-13T15:35:25.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: TEST_scrub_warning: overdue_seconds=993600 2024-09-13T15:35:25.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 993600 + 200 2024-09-13T15:35:25.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph tell 2.0 schedule-scrub 993800 2024-09-13T15:35:25.737 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:35:25.738 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T15:35:25.738 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:35:25.738 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-02T03:30:25.739677+0000" 2024-09-13T15:35:25.738 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:35:25.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-13T15:35:25.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5888: TEST_scrub_warning: '[' 3 = 1 ']' 2024-09-13T15:35:25.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: TEST_scrub_warning: overdue_seconds=993600 2024-09-13T15:35:25.752 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 993600 + 300 2024-09-13T15:35:25.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph tell 3.0 schedule-scrub 993900 2024-09-13T15:35:25.884 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:35:25.884 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T15:35:25.884 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:35:25.884 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-02T03:28:45.885790+0000" 2024-09-13T15:35:25.884 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:35:25.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-13T15:35:25.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5888: TEST_scrub_warning: '[' 4 = 1 ']' 2024-09-13T15:35:25.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: TEST_scrub_warning: overdue_seconds=993600 2024-09-13T15:35:25.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 993600 + 400 2024-09-13T15:35:25.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph tell 4.0 schedule-scrub 994000 2024-09-13T15:35:26.035 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:35:26.036 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T15:35:26.036 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:35:26.036 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-02T03:27:06.037751+0000" 2024-09-13T15:35:26.036 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:35:26.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5886: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-09-13T15:35:26.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5888: TEST_scrub_warning: '[' 5 = 1 ']' 2024-09-13T15:35:26.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: TEST_scrub_warning: overdue_seconds=993600 2024-09-13T15:35:26.050 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 993600 + 500 2024-09-13T15:35:26.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: ceph tell 5.0 schedule-scrub 994100 2024-09-13T15:35:26.171 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:35:26.171 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T15:35:26.172 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:35:26.172 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-02T03:25:26.173692+0000" 2024-09-13T15:35:26.172 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:35:26.186 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:35:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: expr 5 + 5 2024-09-13T15:35:26.188 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: seq 6 10 2024-09-13T15:35:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-13T15:35:26.189 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:35:26.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: '[' 6 = 6 ']' 2024-09-13T15:35:26.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5901: TEST_scrub_warning: overdue_seconds=1900800 2024-09-13T15:35:26.190 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: expr 1900800 + 600 2024-09-13T15:35:26.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: ceph tell 6.0 schedule-deep-scrub 1901400 2024-09-13T15:35:26.314 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:35:26.314 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T15:35:26.314 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:35:26.314 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-08-22T15:23:46.316359+0000" 2024-09-13T15:35:26.314 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:35:26.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-13T15:35:26.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:35:26.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: '[' 7 = 6 ']' 2024-09-13T15:35:26.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: TEST_scrub_warning: overdue_seconds=993600 2024-09-13T15:35:26.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: expr 993600 + 700 2024-09-13T15:35:26.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: ceph tell 7.0 schedule-deep-scrub 994300 2024-09-13T15:35:26.463 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:35:26.463 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T15:35:26.464 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:35:26.464 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-02T03:22:06.465640+0000" 2024-09-13T15:35:26.464 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:35:26.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-13T15:35:26.478 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:35:26.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: '[' 8 = 6 ']' 2024-09-13T15:35:26.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: TEST_scrub_warning: overdue_seconds=993600 2024-09-13T15:35:26.479 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: expr 993600 + 800 2024-09-13T15:35:26.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: ceph tell 8.0 schedule-deep-scrub 994400 2024-09-13T15:35:26.608 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:35:26.608 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T15:35:26.608 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:35:26.608 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-02T03:20:26.609902+0000" 2024-09-13T15:35:26.608 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:35:26.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-13T15:35:26.621 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:35:26.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: '[' 9 = 6 ']' 2024-09-13T15:35:26.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: TEST_scrub_warning: overdue_seconds=993600 2024-09-13T15:35:26.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: expr 993600 + 900 2024-09-13T15:35:26.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: ceph tell 9.0 schedule-deep-scrub 994500 2024-09-13T15:35:26.753 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:35:26.753 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T15:35:26.753 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:35:26.753 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-02T03:18:46.755145+0000" 2024-09-13T15:35:26.753 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:35:26.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5897: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-09-13T15:35:26.767 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: expr 5 + 1 2024-09-13T15:35:26.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5899: TEST_scrub_warning: '[' 10 = 6 ']' 2024-09-13T15:35:26.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: TEST_scrub_warning: overdue_seconds=993600 2024-09-13T15:35:26.769 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: expr 993600 + 1000 2024-09-13T15:35:26.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5905: TEST_scrub_warning: ceph tell 10.0 schedule-deep-scrub 994600 2024-09-13T15:35:26.900 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:35:26.901 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T15:35:26.901 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:35:26.901 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-02T03:17:06.902673+0000" 2024-09-13T15:35:26.901 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:35:26.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5907: TEST_scrub_warning: flush_pg_stats 2024-09-13T15:35:26.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:35:26.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:35:27.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T15:35:27.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:35:27.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:35:27.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:35:27.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836511 2024-09-13T15:35:27.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836511 2024-09-13T15:35:27.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836511' 2024-09-13T15:35:27.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:35:27.363 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836511 2024-09-13T15:35:27.363 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:35:27.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:35:27.365 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836511 2024-09-13T15:35:27.365 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:35:27.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836511 2024-09-13T15:35:27.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-09-13T15:35:27.367 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836511 2024-09-13T15:35:27.367 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:35:27.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836511 2024-09-13T15:35:27.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:35:28.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:35:28.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:35:28.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836511 2024-09-13T15:35:28.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5909: TEST_scrub_warning: ceph health 2024-09-13T15:35:29.332 INFO:tasks.workunit.client.0.smithi112.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 10 pgs not scrubbed in time; 10 pool(s) do not have an application enabled; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2024-09-13T15:35:29.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5910: TEST_scrub_warning: ceph health detail 2024-09-13T15:35:29.700 INFO:tasks.workunit.client.0.smithi112.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 10 pgs not scrubbed in time; 10 pool(s) do not have an application enabled; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2024-09-13T15:35:29.700 INFO:tasks.workunit.client.0.smithi112.stdout:[WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-13T15:35:29.700 INFO:tasks.workunit.client.0.smithi112.stdout:[WRN] PG_NOT_DEEP_SCRUBBED: 5 pgs not deep-scrubbed in time 2024-09-13T15:35:29.700 INFO:tasks.workunit.client.0.smithi112.stdout: pg 10.0 not deep-scrubbed since 2024-09-02T03:17:06.902673+0000 2024-09-13T15:35:29.700 INFO:tasks.workunit.client.0.smithi112.stdout: pg 9.0 not deep-scrubbed since 2024-09-02T03:18:46.755145+0000 2024-09-13T15:35:29.700 INFO:tasks.workunit.client.0.smithi112.stdout: pg 8.0 not deep-scrubbed since 2024-09-02T03:20:26.609902+0000 2024-09-13T15:35:29.701 INFO:tasks.workunit.client.0.smithi112.stdout: pg 7.0 not deep-scrubbed since 2024-09-02T03:22:06.465640+0000 2024-09-13T15:35:29.701 INFO:tasks.workunit.client.0.smithi112.stdout: pg 6.0 not deep-scrubbed since 2024-08-22T15:23:46.316359+0000 2024-09-13T15:35:29.701 INFO:tasks.workunit.client.0.smithi112.stdout:[WRN] PG_NOT_SCRUBBED: 10 pgs not scrubbed in time 2024-09-13T15:35:29.701 INFO:tasks.workunit.client.0.smithi112.stdout: pg 10.0 not scrubbed since 2024-09-02T03:17:06.902673+0000 2024-09-13T15:35:29.701 INFO:tasks.workunit.client.0.smithi112.stdout: pg 9.0 not scrubbed since 2024-09-02T03:18:46.755145+0000 2024-09-13T15:35:29.701 INFO:tasks.workunit.client.0.smithi112.stdout: pg 8.0 not scrubbed since 2024-09-02T03:20:26.609902+0000 2024-09-13T15:35:29.701 INFO:tasks.workunit.client.0.smithi112.stdout: pg 7.0 not scrubbed since 2024-09-02T03:22:06.465640+0000 2024-09-13T15:35:29.701 INFO:tasks.workunit.client.0.smithi112.stdout: pg 6.0 not scrubbed since 2024-08-22T15:23:46.316359+0000 2024-09-13T15:35:29.701 INFO:tasks.workunit.client.0.smithi112.stdout: pg 5.0 not scrubbed since 2024-09-02T03:25:26.173692+0000 2024-09-13T15:35:29.701 INFO:tasks.workunit.client.0.smithi112.stdout: pg 4.0 not scrubbed since 2024-09-02T03:27:06.037751+0000 2024-09-13T15:35:29.701 INFO:tasks.workunit.client.0.smithi112.stdout: pg 3.0 not scrubbed since 2024-09-02T03:28:45.885790+0000 2024-09-13T15:35:29.701 INFO:tasks.workunit.client.0.smithi112.stdout: pg 2.0 not scrubbed since 2024-09-02T03:30:25.739677+0000 2024-09-13T15:35:29.701 INFO:tasks.workunit.client.0.smithi112.stdout: pg 1.0 not scrubbed since 2024-08-22T15:32:05.592292+0000 2024-09-13T15:35:29.701 INFO:tasks.workunit.client.0.smithi112.stdout:[WRN] POOL_APP_NOT_ENABLED: 10 pool(s) do not have an application enabled 2024-09-13T15:35:29.702 INFO:tasks.workunit.client.0.smithi112.stdout: application not enabled on pool 'psr_pool-1' 2024-09-13T15:35:29.702 INFO:tasks.workunit.client.0.smithi112.stdout: application not enabled on pool 'psr_pool-2' 2024-09-13T15:35:29.702 INFO:tasks.workunit.client.0.smithi112.stdout: application not enabled on pool 'psr_pool-3' 2024-09-13T15:35:29.702 INFO:tasks.workunit.client.0.smithi112.stdout: application not enabled on pool 'psr_pool-4' 2024-09-13T15:35:29.702 INFO:tasks.workunit.client.0.smithi112.stdout: application not enabled on pool 'psr_pool-5' 2024-09-13T15:35:29.702 INFO:tasks.workunit.client.0.smithi112.stdout: application not enabled on pool 'psr_pool-6' 2024-09-13T15:35:29.702 INFO:tasks.workunit.client.0.smithi112.stdout: application not enabled on pool 'psr_pool-7' 2024-09-13T15:35:29.702 INFO:tasks.workunit.client.0.smithi112.stdout: application not enabled on pool 'psr_pool-8' 2024-09-13T15:35:29.702 INFO:tasks.workunit.client.0.smithi112.stdout: application not enabled on pool 'psr_pool-9' 2024-09-13T15:35:29.702 INFO:tasks.workunit.client.0.smithi112.stdout: application not enabled on pool 'psr_pool-10' 2024-09-13T15:35:29.702 INFO:tasks.workunit.client.0.smithi112.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-13T15:35:29.702 INFO:tasks.workunit.client.0.smithi112.stdout:[WRN] POOL_NO_REDUNDANCY: 10 pool(s) have no replicas configured 2024-09-13T15:35:29.702 INFO:tasks.workunit.client.0.smithi112.stdout: pool 'psr_pool-1' has no replicas configured 2024-09-13T15:35:29.702 INFO:tasks.workunit.client.0.smithi112.stdout: pool 'psr_pool-2' has no replicas configured 2024-09-13T15:35:29.703 INFO:tasks.workunit.client.0.smithi112.stdout: pool 'psr_pool-3' has no replicas configured 2024-09-13T15:35:29.703 INFO:tasks.workunit.client.0.smithi112.stdout: pool 'psr_pool-4' has no replicas configured 2024-09-13T15:35:29.703 INFO:tasks.workunit.client.0.smithi112.stdout: pool 'psr_pool-5' has no replicas configured 2024-09-13T15:35:29.703 INFO:tasks.workunit.client.0.smithi112.stdout: pool 'psr_pool-6' has no replicas configured 2024-09-13T15:35:29.703 INFO:tasks.workunit.client.0.smithi112.stdout: pool 'psr_pool-7' has no replicas configured 2024-09-13T15:35:29.703 INFO:tasks.workunit.client.0.smithi112.stdout: pool 'psr_pool-8' has no replicas configured 2024-09-13T15:35:29.703 INFO:tasks.workunit.client.0.smithi112.stdout: pool 'psr_pool-9' has no replicas configured 2024-09-13T15:35:29.703 INFO:tasks.workunit.client.0.smithi112.stdout: pool 'psr_pool-10' has no replicas configured 2024-09-13T15:35:29.703 INFO:tasks.workunit.client.0.smithi112.stdout:[WRN] TOO_FEW_OSDS: OSD count 1 < osd_pool_default_size 3 2024-09-13T15:35:29.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5911: TEST_scrub_warning: ceph health 2024-09-13T15:35:29.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5911: TEST_scrub_warning: grep -q ' pgs not deep-scrubbed in time' 2024-09-13T15:35:30.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: grep -q ' pgs not scrubbed in time' 2024-09-13T15:35:30.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_scrub_warning: ceph health 2024-09-13T15:35:30.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: ceph health detail 2024-09-13T15:35:30.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: grep 'not scrubbed since' 2024-09-13T15:35:30.469 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: wc -l 2024-09-13T15:35:30.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_scrub_warning: COUNT=10 2024-09-13T15:35:30.867 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: expr 5+5 2024-09-13T15:35:30.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5919: TEST_scrub_warning: (( 10 != 5 && 10 != 5+5 )) 2024-09-13T15:35:30.869 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: ceph health detail 2024-09-13T15:35:30.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: grep 'not deep-scrubbed since' 2024-09-13T15:35:30.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: wc -l 2024-09-13T15:35:31.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5923: TEST_scrub_warning: COUNT=5 2024-09-13T15:35:31.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5924: TEST_scrub_warning: '[' 5 '!=' 5 ']' 2024-09-13T15:35:31.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T15:35:31.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:35:31.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:35:31.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:35:31.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:35:31.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:35:31.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:35:31.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:35:31.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:35:31.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:35:31.550 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:35:31.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:35:31.552 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:35:31.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:35:31.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:35:31.555 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:35:31.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:35:31.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:35:31.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:35:31.558 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:35:31.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:35:31.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:35:31.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:35:31.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:35:31.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:35:31.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:35:31.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:35:31.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:35:31.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:35:31.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T15:35:31.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T15:35:31.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T15:35:31.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T15:35:31.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:35:31.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:35:31.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:35:31.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:35:31.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:35:31.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:35:31.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:35:31.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:35:31.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:35:31.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:35:31.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:35:31.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:35:31.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:35:31.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:35:31.584 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:35:31.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:35:31.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:35:31.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:35:31.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:35:31.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:35:31.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:35:31.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:35:31.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:35:31.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:35:31.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:35:31.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:35:31.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:35:31.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:35:31.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T15:35:31.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:35:31.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:35:31.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:35:31.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T15:35:31.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:35:31.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T15:35:31.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:35:31.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T15:35:31.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_skip_non_repair_during_recovery td/osd-scrub-repair 2024-09-13T15:35:31.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:136: TEST_skip_non_repair_during_recovery: local dir=td/osd-scrub-repair 2024-09-13T15:35:31.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:137: TEST_skip_non_repair_during_recovery: local poolname=rbd 2024-09-13T15:35:31.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:139: TEST_skip_non_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-13T15:35:31.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T15:35:31.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:35:31.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:35:31.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:35:31.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T15:35:31.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-13T15:35:32.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:35:32.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:35:32.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:35:32.133 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:35:32.133 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:35:32.133 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:35:32.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:35:32.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-13T15:35:32.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:35:32.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:35:32.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:35:32.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:35:32.429 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:35:32.429 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:35:32.429 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:35:32.429 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:35:32.429 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:35:32.430 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:35:32.430 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:35:32.430 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:35:32.430 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:35:32.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:35:32.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T15:35:32.546 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:35:32.546 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:35:32.546 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:35:32.546 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:35:32.547 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:35:32.547 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:35:32.547 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:35:32.547 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:35:32.548 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:35:32.548 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:35:32.548 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:35:32.548 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:35:32.548 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:35:32.549 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T15:35:32.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:140: TEST_skip_non_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-09-13T15:35:32.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T15:35:32.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:35:32.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:35:32.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:35:32.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T15:35:32.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:35:33.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:35:33.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:35:33.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:35:33.598 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:35:33.598 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:35:33.598 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:35:33.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:35:33.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:35:33.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:35:33.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:141: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-13T15:35:33.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:35:33.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:35:33.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:35:33.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:35:33.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:35:33.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:35:33.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:35:33.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:35:33.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:35:33.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:35:33.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:35:33.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:35:33.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:35:33.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:35:33.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:35:33.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:35:33.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:35:33.641 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:35:33.641 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:35:33.641 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:35:33.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:35:33.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:35:33.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:35:33.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:35:33.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:35:33.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:35:33.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:35:33.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:35:33.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:35:33.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:35:33.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:35:33.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:35:33.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-13T15:35:33.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:35:33.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:35:33.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fdb081ed-5c31-4507-9d8a-7dee6e565e78 2024-09-13T15:35:33.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 fdb081ed-5c31-4507-9d8a-7dee6e565e78' 2024-09-13T15:35:33.643 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 fdb081ed-5c31-4507-9d8a-7dee6e565e78 2024-09-13T15:35:33.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:35:33.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDFW+RmZjJXJxAALabCsDrZzNW33tAWDPhA/A== 2024-09-13T15:35:33.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDFW+RmZjJXJxAALabCsDrZzNW33tAWDPhA/A=="}' 2024-09-13T15:35:33.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fdb081ed-5c31-4507-9d8a-7dee6e565e78 -i td/osd-scrub-repair/0/new.json 2024-09-13T15:35:33.963 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:35:33.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T15:35:33.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQDFW+RmZjJXJxAALabCsDrZzNW33tAWDPhA/A== --osd-uuid fdb081ed-5c31-4507-9d8a-7dee6e565e78 2024-09-13T15:35:34.017 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:35:34.015+0000 7f93665447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:35:34.047 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:35:34.047+0000 7f93665447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:35:34.048 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:35:34.047+0000 7f93665447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:35:34.049 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:35:34.047+0000 7f93665447c0 -1 bdev(0x5617dfd25c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:35:34.049 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:35:34.047+0000 7f93665447c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T15:35:41.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T15:35:41.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:35:41.919 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:35:41.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:35:41.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:35:42.318 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:35:42.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:35:42.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-13T15:35:42.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:35:42.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:35:42.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:35:42.353 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:35:42.351+0000 7f7440fb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:35:42.394 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:35:42.391+0000 7f7440fb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:35:42.396 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:35:42.395+0000 7f7440fb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:35:42.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:35:42.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:35:42.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:35:42.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:35:42.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:35:42.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:35:42.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:35:42.644 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:35:42.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:35:42.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:35:42.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:35:43.901 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:35:43.899+0000 7f7440fb47c0 -1 Falling back to public interface 2024-09-13T15:35:43.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:35:43.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:35:43.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:35:43.959 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:35:43.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:35:43.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:35:44.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:35:45.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:35:45.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:35:45.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:35:45.280 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:35:45.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:35:45.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:35:45.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:35:46.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:35:46.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:35:46.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:35:46.602 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:35:46.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:35:46.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:35:46.867 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:35:46.867+0000 7f7440fb47c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:35:46.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:35:47.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:35:47.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:35:47.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:35:47.931 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:35:47.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:35:47.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:35:48.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:35:49.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:35:49.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:35:49.424 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:35:49.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:35:49.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:35:49.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:35:49.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:35:50.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:35:50.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:35:50.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:35:50.926 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:35:50.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:35:50.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:35:51.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:35:52.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:35:52.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:35:52.245 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:35:52.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:35:52.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:35:52.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:35:52.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:35:53.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:35:53.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:35:53.580 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:35:53.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:35:53.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:35:53.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:35:53.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:35:54.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:35:54.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:35:54.962 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:35:54.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:35:54.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:35:54.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:35:55.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:35:56.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:35:56.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:35:56.300 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:35:56.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:35:56.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:35:56.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:35:56.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:35:57.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:35:57.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:35:57.694 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:35:57.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:35:57.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:35:57.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:35:58.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:35:59.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:35:59.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:35:59.214 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:35:59.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:35:59.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:35:59.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:35:59.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:00.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:00.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:00.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:36:00.647 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:36:00.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:00.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:36:01.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:02.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:02.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:02.414 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:36:02.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:36:02.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:02.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:36:02.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:03.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:03.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:03.741 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:36:03.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:36:03.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:03.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:36:04.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:05.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:05.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:05.189 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:36:05.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:36:05.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:05.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:36:05.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:06.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:06.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:06.529 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:36:06.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:36:06.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:06.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:36:07.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:08.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:08.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:08.104 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:36:08.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:36:08.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:08.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:36:08.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:09.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:09.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:09.652 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:36:09.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:36:09.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:09.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:36:10.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:11.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:11.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:11.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:36:11.303 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:36:11.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:11.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:36:11.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:12.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:12.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:12.709 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:36:12.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:36:12.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:12.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:36:13.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:14.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:14.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:14.424 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:36:14.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:36:14.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:14.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:36:14.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:15.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:15.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:15.882 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:36:15.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:36:15.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:15.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:36:16.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:17.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:17.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:17.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:36:17.398 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:36:17.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:17.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:36:17.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:18.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:18.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:18.959 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:36:18.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:36:18.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:18.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:36:19.459 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/430215697,v1:127.0.0.1:6803/430215697] [v2:127.0.0.1:6804/430215697,v1:127.0.0.1:6805/430215697] exists,up fdb081ed-5c31-4507-9d8a-7dee6e565e78 2024-09-13T15:36:19.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:36:19.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:36:19.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:36:19.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:144: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-13T15:36:19.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:36:19.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:36:19.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T15:36:19.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:36:19.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:36:19.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:36:19.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:36:19.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:36:19.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:36:19.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:36:19.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:36:19.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:36:19.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:36:19.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:36:19.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:36:19.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:36:19.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:36:19.463 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:36:19.463 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:36:19.464 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:36:19.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:36:19.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:36:19.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:36:19.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:36:19.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:36:19.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:36:19.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:36:19.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:36:19.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:36:19.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:36:19.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:36:19.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:36:19.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-13T15:36:19.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:36:19.466 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:36:19.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5ab6127e-ae86-4aea-b6b1-edbbceb0cd8d 2024-09-13T15:36:19.467 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 5ab6127e-ae86-4aea-b6b1-edbbceb0cd8d 2024-09-13T15:36:19.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 5ab6127e-ae86-4aea-b6b1-edbbceb0cd8d' 2024-09-13T15:36:19.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:36:19.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDzW+RmopZAHRAAYD988k7N4sLhKQZqaIHbPg== 2024-09-13T15:36:19.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDzW+RmopZAHRAAYD988k7N4sLhKQZqaIHbPg=="}' 2024-09-13T15:36:19.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5ab6127e-ae86-4aea-b6b1-edbbceb0cd8d -i td/osd-scrub-repair/1/new.json 2024-09-13T15:36:19.926 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:36:19.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T15:36:19.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQDzW+RmopZAHRAAYD988k7N4sLhKQZqaIHbPg== --osd-uuid 5ab6127e-ae86-4aea-b6b1-edbbceb0cd8d 2024-09-13T15:36:19.987 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:36:19.986+0000 7f75b09077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:36:20.070 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:36:20.070+0000 7f75b09077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:36:20.072 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:36:20.070+0000 7f75b09077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:36:20.073 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:36:20.070+0000 7f75b09077c0 -1 bdev(0x56365978fc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T15:36:20.073 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:36:20.070+0000 7f75b09077c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T15:36:28.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T15:36:28.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:36:28.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T15:36:28.554 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T15:36:28.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:36:28.962 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:36:28.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T15:36:28.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-13T15:36:28.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:36:28.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:36:28.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:36:28.997 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:36:28.993+0000 7fec872f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:36:29.040 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:36:29.037+0000 7fec872f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:36:29.041 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:36:29.037+0000 7fec872f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:36:29.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T15:36:29.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:36:29.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:36:29.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:36:29.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:36:29.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:29.284 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:36:29.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:36:29.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:29.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:29.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:30.085 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:36:30.081+0000 7fec872f87c0 -1 Falling back to public interface 2024-09-13T15:36:30.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:30.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:30.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:36:30.596 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:36:30.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:30.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:30.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:31.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:31.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:31.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:36:31.965 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:36:31.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:31.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:32.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:33.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:33.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:33.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:36:33.300 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:36:33.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:33.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:33.370 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:36:33.369+0000 7fec872f87c0 -1 osd.1 0 log_to_monitors true 2024-09-13T15:36:33.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:34.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:34.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:34.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:36:34.694 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:36:34.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:34.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:35.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:36.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:36.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:36.184 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:36:36.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:36:36.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:36.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:36.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:37.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:37.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:37.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:36:37.711 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:36:37.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:37.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:38.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:39.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:39.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:39.049 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:36:39.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:36:39.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:39.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:39.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:40.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:40.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:40.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:36:40.547 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:36:40.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:40.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:40.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:41.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:41.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:41.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:36:41.933 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:36:41.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:41.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:42.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:43.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:43.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:43.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:36:43.282 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:36:43.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:43.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:43.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:44.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:44.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:44.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:36:44.800 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:36:44.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:44.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:45.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:46.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:46.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:46.241 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:36:46.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:36:46.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:46.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:46.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:47.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:47.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:47.882 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:36:47.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:36:47.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:47.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:48.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:49.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:49.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:49.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:36:49.272 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:36:49.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:49.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:49.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:50.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:50.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:50.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:36:50.742 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:36:50.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:50.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:51.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:52.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:52.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:52.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:36:52.290 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:36:52.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:52.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:52.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:53.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:53.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:53.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:36:53.697 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:36:53.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:53.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:54.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:55.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:55.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:55.555 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:36:55.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:36:55.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:55.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:56.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:57.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:57.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:57.030 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:36:57.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:36:57.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:57.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:58.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:36:59.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:36:59.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:36:59.121 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:36:59.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:36:59.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:36:59.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:36:59.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:00.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:00.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:00.816 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:37:00.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:37:00.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:00.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:37:01.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:02.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:02.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:02.499 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:37:02.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:37:02.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:02.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:37:02.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:03.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:03.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:03.992 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:37:03.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:37:03.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:03.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:37:04.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:05.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:05.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:05.573 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:37:05.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:37:05.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:05.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:37:06.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:07.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:07.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:07.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:37:07.053 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:37:07.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:07.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:37:07.389 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3712194483,v1:127.0.0.1:6811/3712194483] [v2:127.0.0.1:6812/3712194483,v1:127.0.0.1:6813/3712194483] exists,up 5ab6127e-ae86-4aea-b6b1-edbbceb0cd8d 2024-09-13T15:37:07.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:37:07.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:37:07.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:37:07.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:147: TEST_skip_non_repair_during_recovery: create_rbd_pool 2024-09-13T15:37:07.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-13T15:37:07.729 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' does not exist 2024-09-13T15:37:07.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-13T15:37:07.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-13T15:37:08.196 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'rbd' created 2024-09-13T15:37:08.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:37:09.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-13T15:37:12.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:148: TEST_skip_non_repair_during_recovery: wait_for_clean 2024-09-13T15:37:12.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:37:12.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:37:12.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:37:12.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:37:12.320 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:37:12.320 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:37:12.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:37:12.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:37:12.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:37:12.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:37:12.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:37:12.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:37:12.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:37:12.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:37:12.459 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:37:12.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:37:12.794 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:37:12.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:37:12.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:37:12.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:37:12.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836497 2024-09-13T15:37:12.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836497 2024-09-13T15:37:12.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836497' 2024-09-13T15:37:12.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:37:12.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:37:13.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672971 2024-09-13T15:37:13.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672971 2024-09-13T15:37:13.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836497 1-42949672971' 2024-09-13T15:37:13.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:37:13.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836497 2024-09-13T15:37:13.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:37:13.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:37:13.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836497 2024-09-13T15:37:13.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:37:13.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836497 2024-09-13T15:37:13.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-09-13T15:37:13.118 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836497 2024-09-13T15:37:13.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:37:13.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836496 -lt 21474836497 2024-09-13T15:37:13.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:37:14.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:37:14.429 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:37:14.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836497 2024-09-13T15:37:14.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:37:14.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672971 2024-09-13T15:37:14.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:37:14.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:37:14.767 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672971 2024-09-13T15:37:14.767 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:37:14.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672971 2024-09-13T15:37:14.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-09-13T15:37:14.768 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672971 2024-09-13T15:37:14.769 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:37:15.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672971 -lt 42949672971 2024-09-13T15:37:15.099 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:37:15.100 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:37:15.100 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:37:15.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T15:37:15.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:37:15.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:37:15.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:37:15.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:37:15.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:37:15.484 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:37:15.485 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:37:15.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T15:37:15.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:37:15.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:37:15.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:37:16.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T15:37:16.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:37:16.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:37:16.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:150: TEST_skip_non_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-09-13T15:37:16.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:37:16.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-09-13T15:37:16.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-13T15:37:16.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:37:16.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:37:16.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:37:16.632 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:37:16.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:37:17.159 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:37:17.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:37:17.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:37:17.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-13T15:37:17.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_skip_non_repair_during_recovery: get_not_primary rbd SOMETHING 2024-09-13T15:37:17.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2024-09-13T15:37:17.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-13T15:37:17.760 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2024-09-13T15:37:17.760 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-13T15:37:17.760 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T15:37:17.760 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-13T15:37:17.761 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:37:18.210 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T15:37:18.211 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-09-13T15:37:18.211 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T15:37:18.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_skip_non_repair_during_recovery: scrub_and_not_schedule td/osd-scrub-repair rbd 0 2024-09-13T15:37:18.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:155: scrub_and_not_schedule: local dir=td/osd-scrub-repair 2024-09-13T15:37:18.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:156: scrub_and_not_schedule: local poolname=rbd 2024-09-13T15:37:18.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:157: scrub_and_not_schedule: local osd=0 2024-09-13T15:37:18.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: get_pg rbd SOMETHING 2024-09-13T15:37:18.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-13T15:37:18.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T15:37:18.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-13T15:37:18.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T15:37:18.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: local pg=1.3 2024-09-13T15:37:18.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:163: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-13T15:37:18.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T15:37:18.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:37:18.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:37:18.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T15:37:19.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:163: scrub_and_not_schedule: local last_scrub=2024-09-13T15:37:08.114605+0000 2024-09-13T15:37:19.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:164: scrub_and_not_schedule: ceph tell 1.3 schedule-scrub 2024-09-13T15:37:19.395 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:37:19.395 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T15:37:19.395 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:37:19.395 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-06T15:35:39.396550+0000" 2024-09-13T15:37:19.395 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:37:19.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i=0 )) 2024-09-13T15:37:19.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-09-13T15:37:19.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-13T15:37:19.410 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T15:37:19.410 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:37:19.410 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:37:19.410 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T15:37:19.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-09-13T15:37:08.114605+0000 '>' 2024-09-13T15:37:08.114605+0000 2024-09-13T15:37:19.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-09-13T15:37:20.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-09-13T15:37:20.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-09-13T15:37:20.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-13T15:37:20.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T15:37:20.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:37:20.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:37:20.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T15:37:21.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-09-13T15:37:08.114605+0000 '>' 2024-09-13T15:37:08.114605+0000 2024-09-13T15:37:21.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-09-13T15:37:22.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-09-13T15:37:22.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-09-13T15:37:22.050 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-09-13T15:37:22.050 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-13T15:37:22.050 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:37:22.050 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:37:22.051 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-13T15:37:22.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-09-13T15:37:08.114605+0000 '>' 2024-09-13T15:37:08.114605+0000 2024-09-13T15:37:22.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-09-13T15:37:23.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-09-13T15:37:23.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-09-13T15:37:23.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:179: scrub_and_not_schedule: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-13T15:37:23.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:37:23.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:37:23.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:37:23.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:37:23.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-13T15:37:23.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:37:23.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:37:23.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:37:23.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:37:23.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:37:23.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:37:23.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:37:23.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:37:23.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:37:23.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:37:23.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:37:23.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-13T15:37:23.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:37:23.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:37:23.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:37:23.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:37:23.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:37:23.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-13T15:37:24.209 INFO:tasks.workunit.client.0.smithi112.stdout:_ 2024-09-13T15:37:24.209 INFO:tasks.workunit.client.0.smithi112.stdout:snapset 2024-09-13T15:37:24.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:37:24.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:37:24.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:37:24.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:37:24.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:37:24.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:37:24.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:37:24.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:37:24.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:37:24.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:37:24.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:37:24.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:37:24.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:37:24.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:37:24.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:37:24.501 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:37:24.501 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:37:24.501 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:37:24.503 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:37:24.503 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:37:24.503 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:37:24.503 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:37:24.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:37:24.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:37:24.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:37:24.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:37:24.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:37:24.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:37:24.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:37:24.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:37:24.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:37:24.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:37:24.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:37:24.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:37:24.504 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.0 2024-09-13T15:37:24.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:37:24.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:37:24.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:37:24.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:37:24.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:37:24.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:37:24.538 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:37:24.536+0000 7fea634977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:37:24.541 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:37:24.540+0000 7fea634977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:37:24.542 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:37:24.540+0000 7fea634977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:37:24.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:37:24.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:37:24.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:37:24.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:37:24.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:37:24.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:24.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:37:24.949 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:37:24.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:24.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:37:25.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:25.832 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:37:25.828+0000 7fea634977c0 -1 Falling back to public interface 2024-09-13T15:37:26.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:26.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:26.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:37:26.313 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:37:26.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:26.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:37:26.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:27.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:27.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:27.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:37:27.686 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:37:27.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:27.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:37:28.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:29.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:29.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:29.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:37:29.019 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T15:37:29.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:29.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:37:29.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:29.819 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:37:29.820+0000 7fea634977c0 -1 osd.0 22 log_to_monitors true 2024-09-13T15:37:30.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:30.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:30.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:37:30.416 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T15:37:30.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:30.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:37:30.926 INFO:tasks.workunit.client.0.smithi112.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/2657848527,v1:127.0.0.1:6803/2657848527] [v2:127.0.0.1:6804/2657848527,v1:127.0.0.1:6805/2657848527] exists,up fdb081ed-5c31-4507-9d8a-7dee6e565e78 2024-09-13T15:37:30.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:37:30.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:37:30.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:37:30.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:37:30.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:37:30.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:37:30.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:37:30.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:37:30.929 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:37:30.929 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:37:30.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:37:30.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:37:30.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:37:31.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:37:31.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:37:31.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:37:31.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:37:31.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:37:31.058 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:37:31.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:37:31.673 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:37:31.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:37:31.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:37:31.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:37:31.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149698 2024-09-13T15:37:31.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149698 2024-09-13T15:37:31.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698' 2024-09-13T15:37:31.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:37:31.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:37:32.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672975 2024-09-13T15:37:32.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672975 2024-09-13T15:37:32.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698 1-42949672975' 2024-09-13T15:37:32.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:37:32.053 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149698 2024-09-13T15:37:32.053 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:37:32.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:37:32.055 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149698 2024-09-13T15:37:32.055 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:37:32.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149698 2024-09-13T15:37:32.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-09-13T15:37:32.056 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 111669149698 2024-09-13T15:37:32.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:37:32.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 111669149698 2024-09-13T15:37:32.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:37:33.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:37:33.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:37:33.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149698 -lt 111669149698 2024-09-13T15:37:33.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:37:33.822 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672975 2024-09-13T15:37:33.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:37:33.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:37:33.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672975 2024-09-13T15:37:33.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:37:33.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672975 2024-09-13T15:37:33.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-13T15:37:33.828 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672975 2024-09-13T15:37:33.829 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:37:34.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672976 -lt 42949672975 2024-09-13T15:37:34.150 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:37:34.151 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:37:34.151 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:37:34.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-13T15:37:34.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:37:34.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:37:34.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:37:34.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:37:34.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:37:34.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:37:34.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:37:34.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-13T15:37:34.890 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:37:34.891 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:37:34.891 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:37:35.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-13T15:37:35.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:37:35.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:37:35.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:180: scrub_and_not_schedule: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-13T15:37:35.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:181: scrub_and_not_schedule: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-13T15:37:35.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T15:37:35.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:37:35.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:37:35.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:37:35.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:37:35.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:37:35.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:37:35.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:37:35.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:37:35.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:37:35.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:37:35.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:37:35.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:37:35.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:37:35.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:37:35.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:37:35.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:37:35.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:37:35.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:37:35.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:37:35.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:37:35.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:37:35.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:37:35.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:37:35.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:37:35.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:37:35.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:37:35.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:37:35.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:37:35.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T15:37:35.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T15:37:35.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T15:37:35.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T15:37:35.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:37:35.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:37:35.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:37:35.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:37:35.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:37:35.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:37:35.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:37:35.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:37:35.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:37:35.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:37:35.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:37:35.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:37:35.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:37:35.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:37:35.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:37:35.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:37:35.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:37:35.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:37:35.676 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:37:35.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:37:35.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:37:35.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:37:35.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:37:35.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:37:35.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:37:35.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:37:35.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:37:35.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:37:35.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T15:37:35.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:37:35.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:37:35.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:37:35.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T15:37:35.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:37:35.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T15:37:35.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:37:35.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T15:37:35.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_appends td/osd-scrub-repair 2024-09-13T15:37:35.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:903: TEST_unfound_erasure_coded_appends: unfound_erasure_coded td/osd-scrub-repair false 2024-09-13T15:37:35.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: unfound_erasure_coded: local dir=td/osd-scrub-repair 2024-09-13T15:37:35.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:853: unfound_erasure_coded: local allow_overwrites=false 2024-09-13T15:37:35.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:854: unfound_erasure_coded: local poolname=ecpool 2024-09-13T15:37:35.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:855: unfound_erasure_coded: local payload=ABCDEF 2024-09-13T15:37:35.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-13T15:37:35.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T15:37:35.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:37:35.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:37:35.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:37:35.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T15:37:35.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T15:37:36.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:37:36.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:37:36.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:37:36.247 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:37:36.247 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:37:36.247 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:37:36.247 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:37:36.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T15:37:36.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:37:36.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:37:36.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:37:36.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:37:36.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:37:36.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:37:36.526 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:37:36.526 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:37:36.526 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:37:36.527 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:37:36.527 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:37:36.527 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:37:36.527 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:37:36.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:37:36.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T15:37:36.634 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:37:36.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:37:36.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:37:36.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:37:36.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:37:36.636 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:37:36.636 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:37:36.636 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:37:36.636 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:37:36.636 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:37:36.637 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:37:36.637 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:37:36.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:37:36.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T15:37:36.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-13T15:37:36.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T15:37:36.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:37:36.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:37:36.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:37:36.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T15:37:36.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:37:37.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:37:37.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:37:37.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:37:37.649 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:37:37.649 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:37:37.649 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:37:37.649 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:37:37.650 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:37:37.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:37:37.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: seq 0 3 2024-09-13T15:37:37.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-13T15:37:37.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2024-09-13T15:37:37.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:37:37.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:37:37.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:37:37.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:37:37.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:37:37.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:37:37.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:37:37.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:37:37.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:37:37.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:37:37.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:37:37.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:37:37.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:37:37.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:37:37.691 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:37:37.691 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:37:37.691 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:37:37.691 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:37:37.691 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:37:37.691 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:37:37.693 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:37:37.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:37:37.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:37:37.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:37:37.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:37:37.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:37:37.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:37:37.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:37:37.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:37:37.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:37:37.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:37:37.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:37:37.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:37:37.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:37:37.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:37:37.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ee537d4b-885f-4c4b-8dd7-d584de86123a 2024-09-13T15:37:37.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ee537d4b-885f-4c4b-8dd7-d584de86123a' 2024-09-13T15:37:37.698 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 ee537d4b-885f-4c4b-8dd7-d584de86123a 2024-09-13T15:37:37.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:37:37.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBBXORmaR6AKhAA3Qzm7wwPZAw8px5ydbr11g== 2024-09-13T15:37:37.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBBXORmaR6AKhAA3Qzm7wwPZAw8px5ydbr11g=="}' 2024-09-13T15:37:37.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ee537d4b-885f-4c4b-8dd7-d584de86123a -i td/osd-scrub-repair/0/new.json 2024-09-13T15:37:37.993 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:37:38.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T15:37:38.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBXORmaR6AKhAA3Qzm7wwPZAw8px5ydbr11g== --osd-uuid ee537d4b-885f-4c4b-8dd7-d584de86123a 2024-09-13T15:37:38.043 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:37:38.044+0000 7f37265da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:37:38.077 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:37:38.076+0000 7f37265da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:37:38.078 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:37:38.080+0000 7f37265da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:37:38.079 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:37:38.080+0000 7f37265da7c0 -1 bdev(0x5633f4445c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:37:38.079 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:37:38.080+0000 7f37265da7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T15:37:46.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T15:37:46.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:37:46.183 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:37:46.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:37:46.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:37:46.583 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:37:46.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:37:46.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:37:46.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:37:46.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:37:46.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:37:46.617 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:37:46.615+0000 7fb80d72c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:37:46.666 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:37:46.663+0000 7fb80d72c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:37:46.667 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:37:46.667+0000 7fb80d72c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:37:46.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:37:46.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:37:46.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:37:46.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:37:46.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:37:46.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:46.898 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:37:46.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:37:46.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:46.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:37:47.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:48.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:48.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:48.226 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:37:48.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:37:48.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:48.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:37:48.473 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:37:48.471+0000 7fb80d72c7c0 -1 Falling back to public interface 2024-09-13T15:37:48.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:49.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:49.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:49.548 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:37:49.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:37:49.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:49.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:37:49.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:50.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:50.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:50.877 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:37:50.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:37:50.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:50.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:37:51.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:51.526 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:37:51.523+0000 7fb80d72c7c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:37:52.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:52.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:52.197 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:37:52.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:37:52.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:52.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:37:52.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:53.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:53.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:53.659 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:37:53.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:37:53.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:53.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:37:53.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:54.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:54.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:54.988 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:37:54.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:37:54.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:54.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:37:55.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:56.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:56.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:56.349 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:37:56.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:37:56.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:56.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:37:56.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:57.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:57.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:57.778 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:37:57.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:37:57.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:57.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:37:58.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:37:59.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:37:59.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:37:59.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:37:59.109 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:37:59.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:37:59.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:37:59.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:00.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:00.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:00.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:38:00.432 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:38:00.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:00.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:00.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:01.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:01.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:01.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:38:01.856 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:38:01.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:01.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:02.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:03.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:03.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:03.177 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:38:03.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:38:03.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:03.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:03.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:04.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:04.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:04.633 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:38:04.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:38:04.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:04.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:04.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:05.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:05.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:05.997 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:38:05.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:38:05.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:05.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:06.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:07.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:07.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:07.373 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:38:07.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:38:07.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:07.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:08.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:08.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:08.818 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:38:08.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:38:08.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:08.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:09.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:10.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:10.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:10.342 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:38:10.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:38:10.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:10.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:10.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:11.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:11.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:11.665 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:38:11.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:38:11.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:11.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:12.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:13.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:13.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:13.331 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:38:13.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:38:13.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:13.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:14.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:15.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:15.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:15.004 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:38:15.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:38:15.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:15.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:15.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:16.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:16.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:16.996 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:38:16.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:38:16.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:16.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:17.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:18.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:18.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:18.677 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:38:18.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:38:18.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:18.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:19.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:20.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:20.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:20.534 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:38:20.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:38:20.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:20.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:21.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:22.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:22.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:22.235 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:38:22.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:38:22.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:22.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:22.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:23.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:23.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:23.959 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:38:23.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:38:23.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:23.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:38:24.299 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2093879504,v1:127.0.0.1:6803/2093879504] [v2:127.0.0.1:6804/2093879504,v1:127.0.0.1:6805/2093879504] exists,up ee537d4b-885f-4c4b-8dd7-d584de86123a 2024-09-13T15:38:24.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:38:24.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:38:24.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:38:24.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-13T15:38:24.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2024-09-13T15:38:24.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:38:24.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:38:24.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T15:38:24.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:38:24.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:38:24.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:38:24.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:38:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:38:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:38:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:38:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:38:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:38:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:38:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:38:24.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:38:24.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:38:24.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:38:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:38:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:38:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:38:24.305 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:38:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:38:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:38:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:38:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:38:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:38:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:38:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:38:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:38:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:38:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:38:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:38:24.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:38:24.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:38:24.306 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:38:24.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e5c78fa2-c25d-400b-932f-cc9d1b5c2f37 2024-09-13T15:38:24.307 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 e5c78fa2-c25d-400b-932f-cc9d1b5c2f37 2024-09-13T15:38:24.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e5c78fa2-c25d-400b-932f-cc9d1b5c2f37' 2024-09-13T15:38:24.307 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:38:24.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBwXORmQFtCExAARLJ8XzFkAqf+6iCtJ2I4QA== 2024-09-13T15:38:24.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBwXORmQFtCExAARLJ8XzFkAqf+6iCtJ2I4QA=="}' 2024-09-13T15:38:24.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e5c78fa2-c25d-400b-932f-cc9d1b5c2f37 -i td/osd-scrub-repair/1/new.json 2024-09-13T15:38:24.909 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:38:24.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T15:38:24.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBwXORmQFtCExAARLJ8XzFkAqf+6iCtJ2I4QA== --osd-uuid e5c78fa2-c25d-400b-932f-cc9d1b5c2f37 2024-09-13T15:38:24.964 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:38:24.962+0000 7f7d963657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:38:25.090 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:38:25.086+0000 7f7d963657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:38:25.091 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:38:25.090+0000 7f7d963657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:38:25.092 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:38:25.090+0000 7f7d963657c0 -1 bdev(0x564ce0d27c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T15:38:25.092 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:38:25.090+0000 7f7d963657c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T15:38:33.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T15:38:33.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:38:33.725 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T15:38:33.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T15:38:33.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:38:34.128 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:38:34.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T15:38:34.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:38:34.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:38:34.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:38:34.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:38:34.161 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:38:34.158+0000 7fae1c7e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:38:34.214 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:38:34.210+0000 7fae1c7e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:38:34.215 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:38:34.214+0000 7fae1c7e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:38:34.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T15:38:34.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:38:34.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:38:34.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:38:34.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:38:34.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:34.451 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:38:34.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:38:34.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:34.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:34.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:35.261 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:38:35.258+0000 7fae1c7e87c0 -1 Falling back to public interface 2024-09-13T15:38:35.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:35.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:35.795 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:38:35.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:38:35.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:35.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:36.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:37.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:37.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:37.136 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:38:37.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:38:37.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:37.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:37.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:38.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:38.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:38.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:38:38.487 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:38:38.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:38.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:38.516 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:38:38.486+0000 7fae1c7e87c0 -1 osd.1 0 log_to_monitors true 2024-09-13T15:38:38.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:39.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:39.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:39.832 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:38:39.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:38:39.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:39.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:40.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:41.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:41.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:41.170 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:38:41.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:38:41.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:41.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:41.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:42.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:42.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:42.491 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:42.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:43.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:43.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:43.963 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:38:43.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:38:43.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:43.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:44.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:45.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:45.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:45.465 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:38:45.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:38:45.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:45.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:45.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:46.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:46.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:46.908 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:38:46.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:38:46.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:46.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:47.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:48.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:48.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:48.368 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:38:48.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:38:48.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:48.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:48.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:49.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:49.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:49.985 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:38:49.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:38:49.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:49.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:50.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:51.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:51.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:51.426 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:38:51.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:38:51.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:51.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:51.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:52.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:52.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:52.895 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:38:52.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:38:52.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:52.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:53.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:54.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:54.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:54.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:38:54.387 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:38:54.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:54.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:54.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:55.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:55.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:55.992 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:38:55.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:38:55.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:55.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:56.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:57.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:57.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:57.391 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:38:57.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:38:57.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:57.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:57.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:38:58.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:38:58.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:38:58.832 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:38:58.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:38:58.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:38:58.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:38:59.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:00.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:00.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:00.920 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:39:00.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:39:00.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:00.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:39:01.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:02.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:02.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:02.485 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:39:02.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:39:02.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:02.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:39:03.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:04.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:04.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:04.032 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:39:04.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:39:04.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:04.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:39:04.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:05.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:05.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:05.631 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:39:05.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:39:05.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:05.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:39:06.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:07.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:07.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:07.444 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:39:07.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:39:07.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:07.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:39:08.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:09.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:09.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:09.256 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:39:09.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:39:09.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:09.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:39:09.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:10.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:10.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:10.997 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:39:10.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:39:10.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:10.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:39:11.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:12.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:12.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:12.498 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:39:12.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:39:12.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:12.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:39:12.845 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/678912991,v1:127.0.0.1:6811/678912991] [v2:127.0.0.1:6812/678912991,v1:127.0.0.1:6813/678912991] exists,up e5c78fa2-c25d-400b-932f-cc9d1b5c2f37 2024-09-13T15:39:12.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:39:12.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:39:12.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:39:12.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-13T15:39:12.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2024-09-13T15:39:12.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:39:12.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:39:12.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T15:39:12.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:39:12.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:39:12.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:39:12.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:39:12.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:39:12.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:39:12.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T15:39:12.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T15:39:12.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:39:12.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:39:12.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:39:12.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:39:12.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:39:12.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:39:12.850 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:39:12.850 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:39:12.850 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:39:12.850 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:39:12.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:39:12.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:39:12.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:39:12.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:39:12.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:39:12.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:39:12.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:39:12.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:39:12.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:39:12.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:39:12.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:39:12.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:39:12.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T15:39:12.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:39:12.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=82528972-ba51-407d-96d1-8c9309f33897 2024-09-13T15:39:12.852 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 82528972-ba51-407d-96d1-8c9309f33897 2024-09-13T15:39:12.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 82528972-ba51-407d-96d1-8c9309f33897' 2024-09-13T15:39:12.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:39:12.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCgXORmxms4NBAAbkxyj9mZavs75wYe1lthaQ== 2024-09-13T15:39:12.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCgXORmxms4NBAAbkxyj9mZavs75wYe1lthaQ=="}' 2024-09-13T15:39:12.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 82528972-ba51-407d-96d1-8c9309f33897 -i td/osd-scrub-repair/2/new.json 2024-09-13T15:39:13.338 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:39:13.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T15:39:13.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgXORmxms4NBAAbkxyj9mZavs75wYe1lthaQ== --osd-uuid 82528972-ba51-407d-96d1-8c9309f33897 2024-09-13T15:39:13.396 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:39:13.393+0000 7fa852d0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:39:13.465 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:39:13.461+0000 7fa852d0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:39:13.466 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:39:13.461+0000 7fa852d0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:39:13.467 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:39:13.465+0000 7fa852d0c7c0 -1 bdev(0x556e860e3c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T15:39:13.467 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:39:13.465+0000 7fa852d0c7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T15:39:22.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T15:39:22.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:39:22.365 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T15:39:22.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T15:39:22.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:39:22.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T15:39:22.762 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T15:39:22.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:39:22.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:39:22.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:39:22.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:39:22.796 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:39:22.793+0000 7f44cad3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:39:22.803 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:39:22.801+0000 7f44cad3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:39:22.805 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:39:22.801+0000 7f44cad3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:39:23.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T15:39:23.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:39:23.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:39:23.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:39:23.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:39:23.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:23.107 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:39:23.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:39:23.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:23.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:23.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:39:24.301+0000 7f44cad3e7c0 -1 Falling back to public interface 2024-09-13T15:39:24.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:24.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:24.417 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:39:24.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:39:24.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:24.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:24.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:25.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:25.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:25.753 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:39:25.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:39:25.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:25.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:26.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:27.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:27.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:27.089 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:39:27.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:39:27.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:27.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:27.402 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:39:27.396+0000 7f44cad3e7c0 -1 osd.2 0 log_to_monitors true 2024-09-13T15:39:27.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:28.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:28.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:28.421 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:39:28.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:39:28.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:28.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:29.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:30.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:30.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:30.010 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:39:30.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:39:30.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:30.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:30.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:31.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:31.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:31.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:39:31.354 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:39:31.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:31.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:31.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:32.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:32.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:32.865 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:39:32.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:39:32.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:32.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:33.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:34.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:34.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:34.221 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:39:34.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:39:34.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:34.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:34.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:35.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:35.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:35.708 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:39:35.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:39:35.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:35.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:36.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:37.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:37.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:37.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:39:37.053 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:39:37.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:37.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:37.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:38.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:38.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:38.466 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:39:38.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:39:38.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:38.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:38.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:39.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:39.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:39.803 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:39:39.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:39:39.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:39.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:40.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:41.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:41.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:41.140 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:39:41.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:39:41.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:41.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:41.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:42.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:42.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:42.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:39:42.572 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:39:42.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:42.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:42.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:43.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:43.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:43.989 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:39:43.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:39:43.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:43.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:44.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:45.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:45.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:45.323 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:39:45.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:39:45.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:45.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:45.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:46.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:46.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:46.940 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:39:46.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:39:46.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:46.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:47.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:48.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:48.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:48.547 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:39:48.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:39:48.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:48.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:49.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:50.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:50.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:50.720 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:39:50.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:39:50.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:50.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:51.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:52.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:52.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:52.678 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:39:52.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:39:52.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:52.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:53.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:54.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:54.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:54.625 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:39:54.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:39:54.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:54.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:55.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:56.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:56.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:56.725 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:39:56.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:39:56.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:56.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:57.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:39:58.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:39:58.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:39:58.430 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:39:58.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:39:58.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:39:58.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:39:59.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:00.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:00.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:00.278 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:40:00.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:40:00.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:00.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:40:00.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:01.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:01.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:01.744 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:40:01.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:40:01.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:01.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:40:02.079 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/98891812,v1:127.0.0.1:6819/98891812] [v2:127.0.0.1:6820/98891812,v1:127.0.0.1:6821/98891812] exists,up 82528972-ba51-407d-96d1-8c9309f33897 2024-09-13T15:40:02.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:40:02.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:40:02.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:40:02.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-13T15:40:02.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2024-09-13T15:40:02.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:40:02.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:40:02.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-13T15:40:02.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:40:02.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T15:40:02.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:40:02.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:40:02.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:40:02.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:40:02.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T15:40:02.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T15:40:02.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:40:02.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:40:02.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:40:02.083 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:40:02.083 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:40:02.083 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:40:02.083 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:40:02.083 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:40:02.083 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:40:02.083 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:40:02.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:40:02.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:40:02.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:40:02.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:40:02.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:40:02.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:40:02.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:40:02.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:40:02.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:40:02.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:40:02.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:40:02.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:40:02.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T15:40:02.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:40:02.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=10029408-a3a2-40ab-a4db-a650d1a43575 2024-09-13T15:40:02.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 10029408-a3a2-40ab-a4db-a650d1a43575' 2024-09-13T15:40:02.088 INFO:tasks.workunit.client.0.smithi112.stdout:add osd3 10029408-a3a2-40ab-a4db-a650d1a43575 2024-09-13T15:40:02.088 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:40:02.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDSXORmdjFuBhAALHLARi3fa6A3jZijglwLLw== 2024-09-13T15:40:02.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDSXORmdjFuBhAALHLARi3fa6A3jZijglwLLw=="}' 2024-09-13T15:40:02.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 10029408-a3a2-40ab-a4db-a650d1a43575 -i td/osd-scrub-repair/3/new.json 2024-09-13T15:40:02.535 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:40:02.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-13T15:40:02.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSXORmdjFuBhAALHLARi3fa6A3jZijglwLLw== --osd-uuid 10029408-a3a2-40ab-a4db-a650d1a43575 2024-09-13T15:40:02.590 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:40:02.588+0000 7fbbe3dc07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:40:02.670 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:40:02.668+0000 7fbbe3dc07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:40:02.672 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:40:02.672+0000 7fbbe3dc07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:40:02.672 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:40:02.672+0000 7fbbe3dc07c0 -1 bdev(0x561647b49c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-13T15:40:02.672 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:40:02.672+0000 7fbbe3dc07c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-13T15:40:11.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-13T15:40:11.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:40:11.052 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd3 key to auth repository 2024-09-13T15:40:11.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-13T15:40:11.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:40:11.447 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.3 2024-09-13T15:40:11.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-13T15:40:11.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:40:11.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:40:11.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:40:11.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:40:11.481 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:40:11.479+0000 7fbbbeb967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:40:11.527 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:40:11.527+0000 7fbbbeb967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:40:11.528 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:40:11.527+0000 7fbbbeb967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:40:11.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-13T15:40:11.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:40:11.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T15:40:11.757 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:40:11.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:40:11.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:40:11.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:11.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:40:11.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:11.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:12.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:12.797 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:40:12.795+0000 7fbbbeb967c0 -1 Falling back to public interface 2024-09-13T15:40:13.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:13.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:13.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:40:13.114 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:40:13.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:13.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:13.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:14.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:14.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:14.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:40:14.452 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:40:14.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:14.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:14.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:15.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:15.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:15.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:40:15.825 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:40:15.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:15.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:15.884 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:40:15.883+0000 7fbbbeb967c0 -1 osd.3 0 log_to_monitors true 2024-09-13T15:40:16.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:17.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:17.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:17.197 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:40:17.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:40:17.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:17.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:17.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:18.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:18.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:18.563 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:40:18.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:40:18.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:18.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:19.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:20.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:20.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:20.097 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:40:20.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:40:20.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:20.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:20.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:21.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:21.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:21.437 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:40:21.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:40:21.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:21.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:21.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:22.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:22.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:22.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:40:22.775 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:40:22.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:22.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:23.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:24.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:24.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:24.173 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:40:24.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:40:24.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:24.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:24.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:25.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:25.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:25.610 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:40:25.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:40:25.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:25.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:26.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:27.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:27.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:27.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:40:27.064 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:40:27.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:27.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:27.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:28.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:28.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:28.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:40:28.496 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:40:28.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:28.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:28.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:29.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:29.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:29.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:40:29.814 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:40:29.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:29.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:30.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:31.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:31.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:31.297 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:40:31.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:40:31.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:31.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:31.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:32.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:32.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:32.633 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:40:32.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:40:32.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:32.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:33.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:34.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:34.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:34.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:40:34.213 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:40:34.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:34.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:34.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:35.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:35.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:35.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:40:35.567 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:40:35.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:35.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:35.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:36.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:36.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:36.994 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:40:36.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:40:36.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:36.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:38.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:39.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:39.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:39.004 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:40:39.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:40:39.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:39.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:39.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:40.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:40.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:40.736 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:40:40.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:40:40.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:40.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:41.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:42.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:42.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:42.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:40:42.636 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:40:42.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:42.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:43.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:44.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:44.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:44.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:40:44.734 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:40:44.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:44.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:45.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:46.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:46.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:46.867 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:40:46.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:40:46.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:46.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:47.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:40:48.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:40:48.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:40:48.644 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:40:48.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:40:48.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:40:48.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:40:49.035 INFO:tasks.workunit.client.0.smithi112.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1594703656,v1:127.0.0.1:6827/1594703656] [v2:127.0.0.1:6828/1594703656,v1:127.0.0.1:6829/1594703656] exists,up 10029408-a3a2-40ab-a4db-a650d1a43575 2024-09-13T15:40:49.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:40:49.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:40:49.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:40:49.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: create_ec_pool ecpool false k=2 m=2 2024-09-13T15:40:49.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-13T15:40:49.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-13T15:40:49.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-13T15:40:49.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-13T15:40:49.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-13T15:40:49.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-13T15:40:49.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-13T15:40:50.582 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'ecpool' created 2024-09-13T15:40:50.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:40:51.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-13T15:40:51.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-13T15:40:51.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:40:51.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:40:51.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:40:51.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:40:51.607 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:40:51.607 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:40:51.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:40:51.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:40:51.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:40:51.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:40:51.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:40:51.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:40:51.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:40:51.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:40:51.745 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:40:52.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:40:52.076 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:40:52.076 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:40:52.077 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T15:40:52.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:40:52.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:40:52.077 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:40:52.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836517 2024-09-13T15:40:52.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836517 2024-09-13T15:40:52.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517' 2024-09-13T15:40:52.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:40:52.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:40:52.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672991 2024-09-13T15:40:52.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672991 2024-09-13T15:40:52.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-42949672991' 2024-09-13T15:40:52.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:40:52.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:40:52.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-13T15:40:52.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-13T15:40:52.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-42949672991 2-64424509459' 2024-09-13T15:40:52.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:40:52.558 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T15:40:52.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345928 2024-09-13T15:40:52.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345928 2024-09-13T15:40:52.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-42949672991 2-64424509459 3-85899345928' 2024-09-13T15:40:52.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:40:52.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836517 2024-09-13T15:40:52.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:40:52.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:40:52.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836517 2024-09-13T15:40:52.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:40:52.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836517 2024-09-13T15:40:52.711 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836517 2024-09-13T15:40:52.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-09-13T15:40:52.711 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:40:53.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836517 -lt 21474836517 2024-09-13T15:40:53.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:40:53.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672991 2024-09-13T15:40:53.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:40:53.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:40:53.159 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672991 2024-09-13T15:40:53.159 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:40:53.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672991 2024-09-13T15:40:53.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2024-09-13T15:40:53.161 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672991 2024-09-13T15:40:53.161 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:40:53.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672991 -lt 42949672991 2024-09-13T15:40:53.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:40:53.621 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-13T15:40:53.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:40:53.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:40:53.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-13T15:40:53.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:40:53.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-13T15:40:53.627 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509459 2024-09-13T15:40:53.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-13T15:40:53.628 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:40:54.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509459 2024-09-13T15:40:54.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:40:55.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:40:55.077 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:40:55.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-13T15:40:55.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:40:55.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:40:55.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345928 2024-09-13T15:40:55.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T15:40:55.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345928 2024-09-13T15:40:55.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:40:55.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345928 2024-09-13T15:40:55.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2024-09-13T15:40:55.401 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.3 seq 85899345928 2024-09-13T15:40:55.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T15:40:55.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345928 -lt 85899345928 2024-09-13T15:40:55.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:40:55.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:40:55.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:40:56.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:40:56.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:40:56.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:40:56.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:40:56.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:40:56.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:40:56.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:40:56.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:40:56.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T15:40:56.516 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:40:56.516 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:40:56.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:40:56.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T15:40:56.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T15:40:56.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:40:56.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T15:40:56.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:40:56.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:40:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:40:57.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:40:57.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:40:57.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:40:57.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:40:57.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:40:57.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:40:57.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:40:57.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T15:40:57.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:40:57.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:40:57.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:40:57.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T15:40:57.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T15:40:57.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:40:57.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:40:57.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:40:57.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:40:57.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:40:57.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:40:57.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:40:58.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:40:58.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:40:58.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T15:40:58.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:40:58.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T15:40:58.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:40:58.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:40:58.281 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:40:58.281 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:40:58.281 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:40:58.282 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:40:58.282 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:40:58.282 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:40:58.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:40:58.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:40:58.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:40:58.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:40:59.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:40:59.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:40:59.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:40:59.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-13T15:40:59.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-13T15:40:59.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:40:59.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:40:59.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-13T15:40:59.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:40:59.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:40:59.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:40:59.419 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:40:59.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:40:59.938 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:40:59.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:40:59.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:40:59.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-13T15:41:00.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: get_primary ecpool SOMETHING 2024-09-13T15:41:00.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-13T15:41:00.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T15:41:00.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-13T15:41:00.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:41:00.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: local primary=1 2024-09-13T15:41:00.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: sed -e s/1// 2024-09-13T15:41:00.667 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: get_osds ecpool SOMETHING 2024-09-13T15:41:00.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-13T15:41:00.668 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-13T15:41:00.669 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-13T15:41:00.669 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T15:41:00.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-13T15:41:00.975 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:41:00.975 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:41:00.975 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T15:41:00.976 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 3 2024-09-13T15:41:00.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: osds=('0' '2' '3') 2024-09-13T15:41:00.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: local -a osds 2024-09-13T15:41:00.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: local not_primary_first=0 2024-09-13T15:41:00.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: local not_primary_second=2 2024-09-13T15:41:00.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:871: unfound_erasure_coded: local not_primary_third=3 2024-09-13T15:41:00.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: pids= 2024-09-13T15:41:00.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T15:41:00.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T15:41:00.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T15:41:00.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 453634"' 2024-09-13T15:41:00.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 453634' 2024-09-13T15:41:00.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T15:41:00.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T15:41:00.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T15:41:00.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T15:41:00.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 453636"' 2024-09-13T15:41:00.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 453636' 2024-09-13T15:41:00.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/453637: /' 2024-09-13T15:41:00.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T15:41:00.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T15:41:00.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T15:41:00.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T15:41:00.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 453639"' 2024-09-13T15:41:00.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 453639' 2024-09-13T15:41:00.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/453640: /' 2024-09-13T15:41:00.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: wait_background pids 2024-09-13T15:41:00.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 453634 453636 453639' 2024-09-13T15:41:00.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T15:41:00.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T15:41:00.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 453634 2024-09-13T15:41:00.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T15:41:00.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/453644: /' 2024-09-13T15:41:05.141 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:41:05.141 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:41:05.141 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T15:41:05.142 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:41:05.142 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T15:41:05.142 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:41:05.142 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:41:05.142 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T15:41:05.142 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:41:05.142 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T15:41:05.142 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:41:05.142 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:41:05.142 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:41:05.142 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:41:05.142 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:41:05.143 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:41:05.143 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T15:41:05.143 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:41:05.143 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:41:05.143 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T15:41:05.143 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:41:05.143 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T15:41:05.143 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-13T15:41:05.143 INFO:tasks.workunit.client.0.smithi112.stderr:453644: remove 3#1:eb822e21:::SOMETHING:head# 2024-09-13T15:41:05.143 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T15:41:05.143 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:41:05.143 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:41:05.143 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T15:41:05.144 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:41:05.144 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T15:41:05.144 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:41:05.144 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:41:05.147 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:41:05.147 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:41:05.147 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T15:41:05.147 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T15:41:05.147 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:41:05.147 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:41:05.147 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:41:05.148 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:41:05.148 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:41:05.148 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:41:05.148 INFO:tasks.workunit.client.0.smithi112.stderr:453644: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:41:05.148 INFO:tasks.workunit.client.0.smithi112.stderr:453644: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:41:05.148 INFO:tasks.workunit.client.0.smithi112.stderr:453644: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:41:05.148 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:41:05.148 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:41:05.148 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:41:05.148 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:41:05.148 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:41:05.148 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:41:05.148 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:41:05.149 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:41:05.149 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:41:05.149 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:41:05.149 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:41:05.149 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T15:41:05.149 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T15:41:05.149 INFO:tasks.workunit.client.0.smithi112.stderr:453644: start osd.3 2024-09-13T15:41:05.149 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:41:05.211 INFO:tasks.workunit.client.0.smithi112.stderr:4453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:41:05.211 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:41:05.211 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:41:05.211 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:41:05.211 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T15:41:05.211 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:41:05.211 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:41:05.211 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:41:05.211 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:41:05.211 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:41:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:41:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:41:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:41:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:41:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:41:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:41:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T15:41:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:41:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:41:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:41:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:41:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:41:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-13T15:41:05.213 INFO:tasks.workunit.client.0.smithi112.stderr:453637: remove 1#1:eb822e21:::SOMETHING:head# 2024-09-13T15:41:05.213 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:41:05.213 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:41:05.213 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:41:05.213 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:41:05.213 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:41:05.213 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:41:05.213 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:41:05.213 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:41:05.214 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:41:05.214 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:41:05.214 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:41:05.214 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:41:05.214 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:41:05.214 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:41:05.215 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:41:05.215 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:41:05.215 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:41:05.215 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:41:05.215 INFO:tasks.workunit.client.0.smithi112.stderr:453637: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:41:05.215 INFO:tasks.workunit.client.0.smithi112.stderr:453637: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:41:05.215 INFO:tasks.workunit.client.0.smithi112.stderr:453637: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:41:05.215 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:41:05.216 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:41:05.216 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:41:05.216 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:41:05.216 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:41:05.216 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:41:05.216 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:41:05.216 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:41:05.216 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:41:05.216 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:41:05.216 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:41:05.216 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:41:05.216 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:41:05.217 INFO:tasks.workunit.client.0.smithi112.stderr:453637: start osd.0 2024-09-13T15:41:05.217 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:41:05.365 INFO:tasks.workunit.client.0.smithi112.stderr:4453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:41:05.365 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:41:05.366 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-13T15:41:05.366 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:41:05.366 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T15:41:05.366 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:41:05.366 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:41:05.366 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-13T15:41:05.366 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:41:05.366 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-13T15:41:05.366 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:41:05.366 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:41:05.366 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:41:05.366 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:41:05.367 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:41:05.367 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:41:05.367 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T15:41:05.367 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:41:05.367 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:41:05.367 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-13T15:41:05.367 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:41:05.367 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:41:05.367 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-13T15:41:05.367 INFO:tasks.workunit.client.0.smithi112.stderr:453640: remove 2#1:eb822e21:::SOMETHING:head# 2024-09-13T15:41:05.367 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-13T15:41:05.368 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:41:05.368 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:41:05.368 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T15:41:05.368 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:41:05.368 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:41:05.368 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:41:05.368 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:41:05.369 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:41:05.369 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:41:05.369 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T15:41:05.369 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T15:41:05.369 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:41:05.369 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:41:05.369 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:41:05.369 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:41:05.369 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:41:05.369 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:41:05.369 INFO:tasks.workunit.client.0.smithi112.stderr:453640: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:41:05.370 INFO:tasks.workunit.client.0.smithi112.stderr:453640: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:41:05.370 INFO:tasks.workunit.client.0.smithi112.stderr:453640: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:41:05.370 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:41:05.370 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:41:05.370 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:41:05.370 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:41:05.370 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:41:05.370 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:41:05.371 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:41:05.371 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:41:05.371 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:41:05.371 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:41:05.371 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:41:05.371 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T15:41:05.371 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T15:41:05.371 INFO:tasks.workunit.client.0.smithi112.stderr:453640: start osd.2 2024-09-13T15:41:05.371 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:41:10.838 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:41:10.838 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:41:10.838 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:41:10.839 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:41:10.839 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:41:10.839 INFO:tasks.workunit.client.0.smithi112.stderr:453637: 2024-09-13T15:41:05.238+0000 7f761d0977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:41:10.839 INFO:tasks.workunit.client.0.smithi112.stderr:453637: 2024-09-13T15:41:05.242+0000 7f761d0977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:41:10.839 INFO:tasks.workunit.client.0.smithi112.stderr:453637: 2024-09-13T15:41:05.242+0000 7f761d0977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:41:10.839 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:41:10.839 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:41:10.839 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:41:10.839 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:41:10.839 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:41:10.839 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:10.839 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:41:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:453637: 0 2024-09-13T15:41:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:453637: 2024-09-13T15:41:06.818+0000 7f761d0977c0 -1 Falling back to public interface 2024-09-13T15:41:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:41:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:453637: 1 2024-09-13T15:41:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:10.840 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:41:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:453637: 2 2024-09-13T15:41:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:41:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:453637: 3 2024-09-13T15:41:10.841 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:10.842 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:10.842 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:10.879 INFO:tasks.workunit.client.0.smithi112.stderr:453637: 2024-09-13T15:41:10.834+0000 7f761d0977c0 -53644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T15:41:10.879 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T15:41:10.880 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:41:10.880 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:41:10.880 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:41:10.880 INFO:tasks.workunit.client.0.smithi112.stderr:453644: 2024-09-13T15:41:05.182+0000 7f0c772107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:41:10.880 INFO:tasks.workunit.client.0.smithi112.stderr:453644: 2024-09-13T15:41:05.186+0000 7f0c772107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:41:10.880 INFO:tasks.workunit.client.0.smithi112.stderr:453644: 2024-09-13T15:41:05.186+0000 7f0c772107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:41:10.880 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T15:41:10.880 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:41:10.880 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T15:41:10.880 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:41:10.880 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:41:10.881 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:10.881 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:41:10.881 INFO:tasks.workunit.client.0.smithi112.stderr:453644: 0 2024-09-13T15:41:10.881 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:10.881 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:41:10.881 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:10.881 INFO:tasks.workunit.client.0.smithi112.stderr:453644: 2024-09-13T15:41:06.710+0000 7f0c772107c0 -1 Falling back to public interface 2024-09-13T15:41:10.881 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:10.881 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:10.881 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:41:10.881 INFO:tasks.workunit.client.0.smithi112.stderr:453644: 1 2024-09-13T15:41:10.882 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:10.882 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:41:10.882 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:10.882 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:10.882 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:10.882 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:41:10.882 INFO:tasks.workunit.client.0.smithi112.stderr:453644: 2 2024-09-13T15:41:10.882 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:10.882 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:41:10.882 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:10.882 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:10.882 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:10.883 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:41:10.883 INFO:tasks.workunit.client.0.smithi112.stderr:453644: 3 2024-09-13T15:41:10.883 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:10.883 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:41:10.883 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:11.044 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/sta53640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-13T15:41:11.044 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T15:41:11.044 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:41:11.044 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:41:11.044 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:41:11.044 INFO:tasks.workunit.client.0.smithi112.stderr:453640: 2024-09-13T15:41:05.410+0000 7f0f5452f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:41:11.044 INFO:tasks.workunit.client.0.smithi112.stderr:453640: 2024-09-13T15:41:05.410+0000 7f0f5452f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:41:11.044 INFO:tasks.workunit.client.0.smithi112.stderr:453640: 2024-09-13T15:41:05.414+0000 7f0f5452f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:41:11.045 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T15:41:11.045 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:41:11.045 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:41:11.045 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:41:11.045 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:41:11.045 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:11.045 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:41:11.045 INFO:tasks.workunit.client.0.smithi112.stderr:453640: 0 2024-09-13T15:41:11.045 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:11.045 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:41:11.045 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:11.045 INFO:tasks.workunit.client.0.smithi112.stderr:453640: 2024-09-13T15:41:06.742+0000 7f0f5452f7c0 -1 Falling back to public interface 2024-09-13T15:41:11.045 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:11.046 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:11.046 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:41:11.046 INFO:tasks.workunit.client.0.smithi112.stderr:453640: 1 2024-09-13T15:41:11.046 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:11.046 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:41:11.046 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:11.046 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:11.046 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:11.046 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:41:11.046 INFO:tasks.workunit.client.0.smithi112.stderr:453640: 2 2024-09-13T15:41:11.046 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:11.046 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:41:11.047 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:11.047 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:11.047 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:11.047 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:41:11.047 INFO:tasks.workunit.client.0.smithi112.stderr:453640: 3 2024-09-13T15:41:11.047 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:11.047 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:41:11.047 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:13.883 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/sta1 osd.0 28 log_to_monitors true 2024-09-13T15:41:13.884 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:13.884 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:13.884 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:41:13.884 INFO:tasks.workunit.client.0.smithi112.stderr:453637: 4 2024-09-13T15:41:13.884 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:13.884 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:13.884 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:13.885 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:13.885 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:13.885 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:41:13.885 INFO:tasks.workunit.client.0.smithi112.stderr:453637: 5 2024-09-13T15:41:13.885 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:13.885 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:13.885 INFO:tasks.workunit.client.0.smithi112.stderr:453637: osd.0 up in weight 1 up_from 32 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6826/1271538276,v1:127.0.0.1:6827/1271538276] [v2:127.0.0.1:6828/1271538276,v1:127.0.0.1:6829/1271538276] exists,up ee537d4b-885f-4c4b-8dd7-d584de86123a 2024-09-13T15:41:13.885 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:41:13.885 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:41:13.885 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:41:13.885 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:41:13.885 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:41:13.886 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:41:13.886 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:41:13.886 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:41:13.886 INFO:tasks.workunit.client.0.smithi112.stderr:453637: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:41:13.886 INFO:tasks.workunit.client.0.smithi112.stderr:453637: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:41:13.886 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:41:13.886 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:41:13.886 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:41:13.886 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:41:13.887 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:41:13.887 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:41:13.887 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:41:13.887 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:41:13.887 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:41:13.887 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:41:13.887 INFO:tasks.workunit.client.0.smithi112.stderr:453637: 1 2024-09-13T15:41:13.887 INFO:tasks.workunit.client.0.smithi112.stderr:453637: 2 2024-09-13T15:41:13.887 INFO:tasks.workunit.client.0.smithi112.stderr:453637: 3' 2024-09-13T15:41:13.887 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:41:13.887 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:41:13.892 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubundalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:13.892 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:13.892 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:41:13.892 INFO:tasks.workunit.client.0.smithi112.stderr:453640: 4 2024-09-13T15:41:13.892 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:41:13.892 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:13.892 INFO:tasks.workunit.client.0.smithi112.stderr:453640: 2024-09-13T15:41:11.458+0000 7f0f5452f7c0 -1 osd.2 28 log_to_monitors true 2024-09-13T15:41:13.893 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:13.893 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:13.893 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:13.893 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:41:13.893 INFO:tasks.workunit.client.0.smithi112.stderr:453640: 5 2024-09-13T15:41:13.893 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:13.893 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:41:13.893 INFO:tasks.workunit.client.0.smithi112.stderr:453640: osd.2 up in weight 1 up_from 33 up_thru 0 down_at 29 last_clean_interval [15,28) [v2:127.0.0.1:6818/4176304851,v1:127.0.0.1:6819/4176304851] [v2:127.0.0.1:6820/4176304851,v1:127.0.0.1:6821/4176304851] exists,up 82528972-ba51-407d-96d1-8c9309f33897 2024-09-13T15:41:13.893 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:41:13.893 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:41:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:41:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:41:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:41:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:41:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:41:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:41:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:453640: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:41:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:453640: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:41:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:41:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:41:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:41:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:41:13.894 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:41:13.895 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:41:13.895 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:41:13.895 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:41:13.895 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:41:13.895 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:41:13.895 INFO:tasks.workunit.client.0.smithi112.stderr:453640: 1 2024-09-13T15:41:13.895 INFO:tasks.workunit.client.0.smithi112.stderr:453640: 2 2024-09-13T15:41:13.895 INFO:tasks.workunit.client.0.smithi112.stderr:453640: 3' 2024-09-13T15:41:13.896 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:41:13.896 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:41:13.896 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubndalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:13.896 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:13.896 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:41:13.896 INFO:tasks.workunit.client.0.smithi112.stderr:453644: 4 2024-09-13T15:41:13.896 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:13.896 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:41:13.896 INFO:tasks.workunit.client.0.smithi112.stderr:453644: 2024-09-13T15:41:11.142+0000 7f0c772107c0 -1 osd.3 28 log_to_monitors true 2024-09-13T15:41:13.896 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:13.896 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:13.897 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:13.897 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:41:13.897 INFO:tasks.workunit.client.0.smithi112.stderr:453644: 5 2024-09-13T15:41:13.897 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:13.897 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:41:13.897 INFO:tasks.workunit.client.0.smithi112.stderr:453644: osd.3 up in weight 1 up_from 33 up_thru 0 down_at 29 last_clean_interval [20,28) [v2:127.0.0.1:6802/1709099860,v1:127.0.0.1:6803/1709099860] [v2:127.0.0.1:6804/1709099860,v1:127.0.0.1:6805/1709099860] exists,up 10029408-a3a2-40ab-a4db-a650d1a43575 2024-09-13T15:41:13.897 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:41:13.897 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:41:13.897 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:41:13.897 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:41:13.897 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:41:13.897 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:41:13.897 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:41:13.898 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:41:13.898 INFO:tasks.workunit.client.0.smithi112.stderr:453644: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:41:13.898 INFO:tasks.workunit.client.0.smithi112.stderr:453644: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:41:13.898 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:41:13.898 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:41:13.898 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:41:13.898 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:41:13.898 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:41:13.898 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:41:13.898 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:41:13.898 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:41:13.898 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:41:13.898 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:41:13.899 INFO:tasks.workunit.client.0.smithi112.stderr:453644: 1 2024-09-13T15:41:13.899 INFO:tasks.workunit.client.0.smithi112.stderr:453644: 2 2024-09-13T15:41:13.899 INFO:tasks.workunit.client.0.smithi112.stderr:453644: 3' 2024-09-13T15:41:13.899 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:41:13.899 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:41:14.971 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:41:14.971 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953475 2024-09-13T15:41:14.971 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953475 2024-09-13T15:41:14.971 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953475' 2024-09-13T15:41:14.972 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:41:14.972 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:41:14.972 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672997 2024-09-13T15:41:14.972 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672997 2024-09-13T15:41:14.972 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953475 1-42949672997' 2024-09-13T15:41:14.972 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:41:14.973 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:41:14.973 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920771 2024-09-13T15:41:14.973 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920771 2024-09-13T15:41:14.973 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953475 1-42949672997 2-141733920771' 2024-09-13T15:41:14.973 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:41:14.973 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T15:41:14.973 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920771 2024-09-13T15:41:14.973 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920771 2024-09-13T15:41:14.973 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953475 1-42949672997 2-141733920771 3-141733920771' 2024-09-13T15:41:14.973 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:41:14.973 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-137438953475 2024-09-13T15:41:14.973 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:41:14.974 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:41:14.974 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-137438953475 2024-09-13T15:41:14.974 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:41:14.974 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953475 2024-09-13T15:41:14.974 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 137438953475' 2024-09-13T15:41:14.974 INFO:tasks.workunit.client.0.smithi112.stderr:453644: waiting osd.0 seq 137438953475 2024-09-13T15:41:14.974 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:41:14.974 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953476 -lt 137438953475 2024-09-13T15:41:14.974 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:41:14.974 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672997 2024-09-13T15:41:14.974 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:41:14.975 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:41:14.975 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpentu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:41:14.975 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953474 2024-09-13T15:41:14.975 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953474 2024-09-13T15:41:14.975 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953474' 2024-09-13T15:41:14.975 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:41:14.975 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:41:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672998 2024-09-13T15:41:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672998 2024-09-13T15:41:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953474 1-42949672998' 2024-09-13T15:41:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:41:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:41:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920770 2024-09-13T15:41:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920770 2024-09-13T15:41:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953474 1-42949672998 2-141733920770' 2024-09-13T15:41:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:41:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T15:41:14.976 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920770 2024-09-13T15:41:14.977 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920770 2024-09-13T15:41:14.977 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953474 1-42949672998 2-141733920770 3-141733920770' 2024-09-13T15:41:14.977 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:41:14.977 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-137438953474 2024-09-13T15:41:14.977 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:41:14.977 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:41:14.977 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-137438953474 2024-09-13T15:41:14.978 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:41:14.978 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953474 2024-09-13T15:41:14.978 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 137438953474' 2024-09-13T15:41:14.978 INFO:tasks.workunit.client.0.smithi112.stderr:453637: waiting osd.0 seq 137438953474 2024-09-13T15:41:14.978 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:41:14.978 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953476 -lt 137438953474 2024-09-13T15:41:14.978 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:41:14.978 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672998 2024-09-13T15:41:14.978 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:41:14.979 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:41:14.979 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:41:14.979 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953476 2024-09-13T15:41:14.979 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953476 2024-09-13T15:41:14.979 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953476' 2024-09-13T15:41:14.979 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:41:14.979 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:41:14.979 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672996 2024-09-13T15:41:14.980 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672996 2024-09-13T15:41:14.980 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953476 1-42949672996' 2024-09-13T15:41:14.980 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:41:14.980 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:41:14.981 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920772 2024-09-13T15:41:14.981 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920772 2024-09-13T15:41:14.981 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953476 1-42949672996 2-141733920772' 2024-09-13T15:41:14.981 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:41:14.981 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T15:41:14.981 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920772 2024-09-13T15:41:14.982 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920772 2024-09-13T15:41:14.982 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-137438953476 1-42949672996 2-141733920772 3-141733920772' 2024-09-13T15:41:14.982 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:41:14.982 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-137438953476 2024-09-13T15:41:14.982 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:41:14.982 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:41:14.983 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-137438953476 2024-09-13T15:41:14.983 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:41:14.983 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953476 2024-09-13T15:41:14.983 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 137438953476' 2024-09-13T15:41:14.983 INFO:tasks.workunit.client.0.smithi112.stderr:453640: waiting osd.0 seq 137438953476 2024-09-13T15:41:14.983 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:41:14.984 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953476 -lt 137438953476 2024-09-13T15:41:14.984 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:41:14.984 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672996 2024-09-13T15:41:14.984 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:41:14.984 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:41:17.729 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpes.sh:2220: flush_pg_stats: echo 1-42949672998 2024-09-13T15:41:17.729 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:41:17.729 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672998 2024-09-13T15:41:17.729 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-09-13T15:41:17.730 INFO:tasks.workunit.client.0.smithi112.stderr:453637: waiting osd.1 seq 42949672998 2024-09-13T15:41:17.730 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:41:17.730 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672998 -lt 42949672998 2024-09-13T15:41:17.730 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:41:17.730 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:41:17.730 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920770 2024-09-13T15:41:17.730 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:41:17.730 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920770 2024-09-13T15:41:17.730 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:41:17.730 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920770 2024-09-13T15:41:17.730 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920770' 2024-09-13T15:41:17.730 INFO:tasks.workunit.client.0.smithi112.stderr:453637: waiting osd.2 seq 141733920770 2024-09-13T15:41:17.730 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:41:17.730 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920770 2024-09-13T15:41:17.731 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:41:17.731 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-141733920770 2024-09-13T15:41:17.731 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:41:17.731 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T15:41:17.731 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-141733920770 2024-09-13T15:41:17.731 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:41:17.731 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920770 2024-09-13T15:41:17.731 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 141733920770' 2024-09-13T15:41:17.731 INFO:tasks.workunit.client.0.smithi112.stderr:453637: waiting osd.3 seq 141733920770 2024-09-13T15:41:17.731 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T15:41:17.731 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 141733920770 2024-09-13T15:41:17.731 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:41:17.731 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:41:17.732 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T15:41:17.732 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920770 2024-09-13T15:41:17.732 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:41:17.732 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:41:17.732 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:41:17.732 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:41:17.815 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /homers.sh:2220: flush_pg_stats: echo 1-42949672996 2024-09-13T15:41:17.815 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:41:17.815 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672996 2024-09-13T15:41:17.815 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-09-13T15:41:17.816 INFO:tasks.workunit.client.0.smithi112.stderr:453640: waiting osd.1 seq 42949672996 2024-09-13T15:41:17.816 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:41:17.816 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672998 -lt 42949672996 2024-09-13T15:41:17.816 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:41:17.816 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920772 2024-09-13T15:41:17.816 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:41:17.816 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:41:17.817 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920772 2024-09-13T15:41:17.817 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:41:17.817 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920772 2024-09-13T15:41:17.817 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920772' 2024-09-13T15:41:17.817 INFO:tasks.workunit.client.0.smithi112.stderr:453640: waiting osd.2 seq 141733920772 2024-09-13T15:41:17.817 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:41:17.817 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920772 2024-09-13T15:41:17.817 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:41:17.818 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-141733920772 2024-09-13T15:41:17.818 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:41:17.818 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T15:41:17.818 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-141733920772 2024-09-13T15:41:17.818 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:41:17.818 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920772 2024-09-13T15:41:17.818 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 141733920772' 2024-09-13T15:41:17.818 INFO:tasks.workunit.client.0.smithi112.stderr:453640: waiting osd.3 seq 141733920772 2024-09-13T15:41:17.818 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T15:41:17.818 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 141733920772 2024-09-13T15:41:17.818 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:41:17.819 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:41:17.819 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T15:41:17.819 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920772 2024-09-13T15:41:17.819 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:41:17.819 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:41:17.819 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:41:17.819 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:41:17.892 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /homrs.sh:2220: flush_pg_stats: echo 1-42949672997 2024-09-13T15:41:17.892 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:41:17.892 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672997 2024-09-13T15:41:17.892 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2024-09-13T15:41:17.892 INFO:tasks.workunit.client.0.smithi112.stderr:453644: waiting osd.1 seq 42949672997 2024-09-13T15:41:17.892 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:41:17.892 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672998 -lt 42949672997 2024-09-13T15:41:17.892 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:41:17.892 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920771 2024-09-13T15:41:17.892 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:41:17.892 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:41:17.893 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920771 2024-09-13T15:41:17.893 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:41:17.893 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920771 2024-09-13T15:41:17.893 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2024-09-13T15:41:17.893 INFO:tasks.workunit.client.0.smithi112.stderr:453644: waiting osd.2 seq 141733920771 2024-09-13T15:41:17.893 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:41:17.893 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920771 2024-09-13T15:41:17.893 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:41:17.893 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-141733920771 2024-09-13T15:41:17.893 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:41:17.894 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T15:41:17.894 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-141733920771 2024-09-13T15:41:17.894 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:41:17.894 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920771 2024-09-13T15:41:17.894 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 141733920771' 2024-09-13T15:41:17.894 INFO:tasks.workunit.client.0.smithi112.stderr:453644: waiting osd.3 seq 141733920771 2024-09-13T15:41:17.894 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T15:41:17.894 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 141733920771 2024-09-13T15:41:17.894 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:41:17.894 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:41:17.894 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T15:41:17.894 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920771 2024-09-13T15:41:17.895 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:41:17.895 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:41:17.895 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:41:17.895 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:41:18.447 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /hom/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:41:18.447 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:41:18.447 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:41:18.447 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:41:18.447 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:41:18.448 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:41:18.448 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:41:18.448 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:41:18.448 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:41:18.448 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:41:18.448 INFO:tasks.workunit.client.0.smithi112.stderr:453637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:41:18.448 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:41:18.448 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:41:18.448 INFO:tasks.workunit.client.0.smithi112.stderr:453637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:41:18.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T15:41:18.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T15:41:18.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 453636 2024-09-13T15:41:18.524 INFO:tasks.workunit.client.0.smithi112.stderr:e/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:41:18.525 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:41:18.525 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:41:18.525 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:41:18.525 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:41:18.525 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:41:18.525 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:41:18.525 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:41:18.525 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:41:18.525 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:41:18.525 INFO:tasks.workunit.client.0.smithi112.stderr:453640: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:41:18.526 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:41:18.526 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:41:18.526 INFO:tasks.workunit.client.0.smithi112.stderr:453640: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:41:18.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T15:41:18.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T15:41:18.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 453639 2024-09-13T15:41:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:e/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:41:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:41:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:41:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:41:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:41:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:41:18.573 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:41:18.574 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:41:18.574 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:41:18.574 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:41:18.574 INFO:tasks.workunit.client.0.smithi112.stderr:453644: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:41:18.574 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:41:18.574 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:41:18.574 INFO:tasks.workunit.client.0.smithi112.stderr:453644: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:41:18.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T15:41:18.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T15:41:18.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T15:41:18.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T15:41:18.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: return_code=0 2024-09-13T15:41:18.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:882: unfound_erasure_coded: '[' 0 -ne 0 ']' 2024-09-13T15:41:18.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: get_pg ecpool SOMETHING 2024-09-13T15:41:18.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T15:41:18.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T15:41:18.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T15:41:18.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T15:41:18.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: local pg=1.0 2024-09-13T15:41:18.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: unfound_erasure_coded: repair 1.0 2024-09-13T15:41:18.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-13T15:41:18.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-13T15:41:18.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:41:18.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:41:18.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:41:18.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:41:19.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T15:40:50.399116+0000 2024-09-13T15:41:19.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-13T15:41:19.600 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-09-13T15:41:19.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-13T15:40:50.399116+0000 2024-09-13T15:41:19.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T15:41:19.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:40:50.399116+0000 2024-09-13T15:41:19.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:41:19.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:41:19.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:41:19.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:41:19.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:41:19.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:41:19.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:41:19.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:41:19.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:40:50.399116+0000 '>' 2024-09-13T15:40:50.399116+0000 2024-09-13T15:41:19.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:41:20.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:41:20.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:41:20.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:41:20.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:41:20.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:41:20.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:41:20.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:41:21.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:40:50.399116+0000 '>' 2024-09-13T15:40:50.399116+0000 2024-09-13T15:41:21.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:41:22.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:41:22.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:41:22.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:41:22.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:41:22.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:41:22.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:41:22.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:41:22.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:41:19.992324+0000 '>' 2024-09-13T15:40:50.399116+0000 2024-09-13T15:41:22.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:41:22.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T15:41:24.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: seq 1 60 2024-09-13T15:41:24.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: for f in `seq 1 60` 2024-09-13T15:41:24.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: ceph -s 2024-09-13T15:41:24.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-13T15:41:24.951 INFO:tasks.workunit.client.0.smithi112.stdout: 1/1 objects unfound (100.000%) 2024-09-13T15:41:24.951 INFO:tasks.workunit.client.0.smithi112.stdout: 1/1 objects unfound (100.000%) 2024-09-13T15:41:24.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: break 2024-09-13T15:41:24.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: grep '4 up' 2024-09-13T15:41:24.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: ceph -s 2024-09-13T15:41:25.361 INFO:tasks.workunit.client.0.smithi112.stdout: osd: 4 osds: 4 up (since 13s), 4 in (since 82s) 2024-09-13T15:41:25.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: ceph -s 2024-09-13T15:41:25.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: grep '4 in' 2024-09-13T15:41:25.789 INFO:tasks.workunit.client.0.smithi112.stdout: osd: 4 osds: 4 up (since 13s), 4 in (since 83s) 2024-09-13T15:41:25.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: ceph -s 2024-09-13T15:41:25.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-13T15:41:26.237 INFO:tasks.workunit.client.0.smithi112.stdout: 1/1 objects unfound (100.000%) 2024-09-13T15:41:26.237 INFO:tasks.workunit.client.0.smithi112.stdout: 1/1 objects unfound (100.000%) 2024-09-13T15:41:26.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T15:41:26.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:41:26.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:41:26.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:41:26.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:41:26.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:41:26.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:41:26.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:41:26.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:41:26.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:41:26.553 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:41:26.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:41:26.556 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:41:26.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:41:26.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:41:26.557 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:41:26.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:41:26.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:41:26.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:41:26.559 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:41:26.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:41:26.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:41:26.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:41:26.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:41:26.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:41:26.591 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:41:26.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:41:26.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:41:26.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:41:26.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-13T15:41:26.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-13T15:41:26.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-13T15:41:26.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-13T15:41:26.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:41:26.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:41:26.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:41:26.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:41:26.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:41:26.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:41:26.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:41:26.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:41:26.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:41:26.596 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:41:26.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:41:26.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:41:26.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:41:26.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:41:26.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:41:26.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:41:26.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:41:26.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:41:26.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:41:26.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:41:26.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:41:26.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:41:26.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:41:26.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:41:26.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:41:26.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:41:26.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:41:26.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:41:26.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-13T15:41:26.609 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:41:26.609 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:41:26.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:41:26.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.63282 2024-09-13T15:41:26.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:41:26.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T15:41:26.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:41:26.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-13T15:41:26.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_overwrites td/osd-scrub-repair 2024-09-13T15:41:26.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: TEST_unfound_erasure_coded_overwrites: '[' true = true ']' 2024-09-13T15:41:26.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:908: TEST_unfound_erasure_coded_overwrites: unfound_erasure_coded td/osd-scrub-repair true 2024-09-13T15:41:26.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: unfound_erasure_coded: local dir=td/osd-scrub-repair 2024-09-13T15:41:26.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:853: unfound_erasure_coded: local allow_overwrites=true 2024-09-13T15:41:26.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:854: unfound_erasure_coded: local poolname=ecpool 2024-09-13T15:41:26.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:855: unfound_erasure_coded: local payload=ABCDEF 2024-09-13T15:41:26.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-13T15:41:26.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-13T15:41:26.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:41:26.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:41:26.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:41:26.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-13T15:41:26.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-13T15:41:27.143 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:41:27.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:41:27.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:41:27.144 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:41:27.144 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:41:27.144 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:41:27.145 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:41:27.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T15:41:27.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:41:27.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:41:27.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:41:27.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:41:27.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:41:27.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:41:27.401 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:41:27.402 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:41:27.402 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:41:27.403 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:41:27.403 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:41:27.403 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:41:27.403 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:41:27.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:41:27.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get fsid 2024-09-13T15:41:27.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:41:27.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:41:27.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:41:27.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:41:27.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:41:27.523 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:41:27.523 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:41:27.523 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:41:27.523 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:41:27.524 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:41:27.524 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:41:27.524 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.63282/ceph-mon.a.asok 2024-09-13T15:41:27.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:41:27.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.63282/ceph-mon.a.asok config get mon_host 2024-09-13T15:41:27.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-13T15:41:27.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-13T15:41:27.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:41:27.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:41:27.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:41:27.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-13T15:41:27.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:41:28.558 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:41:28.558 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:41:28.559 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:41:28.559 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:41:28.559 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:41:28.559 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:41:28.559 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:41:28.560 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:41:28.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:41:28.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: seq 0 3 2024-09-13T15:41:28.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-13T15:41:28.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2024-09-13T15:41:28.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:41:28.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:41:28.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:41:28.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:41:28.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:41:28.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:41:28.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:41:28.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:41:28.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:41:28.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:41:28.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:41:28.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:41:28.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:41:28.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:41:28.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:41:28.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:41:28.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:41:28.602 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:41:28.602 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:41:28.602 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:41:28.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:41:28.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:41:28.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:41:28.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:41:28.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:41:28.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:41:28.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:41:28.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:41:28.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:41:28.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:41:28.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:41:28.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:41:28.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:41:28.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:41:28.603 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:41:28.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=97386219-fa2e-46b4-ae17-d7c22d322b16 2024-09-13T15:41:28.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 97386219-fa2e-46b4-ae17-d7c22d322b16' 2024-09-13T15:41:28.604 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 97386219-fa2e-46b4-ae17-d7c22d322b16 2024-09-13T15:41:28.604 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:41:28.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAoXeRmApghJRAAN6vT5798KykrASFBjazlwA== 2024-09-13T15:41:28.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAoXeRmApghJRAAN6vT5798KykrASFBjazlwA=="}' 2024-09-13T15:41:28.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 97386219-fa2e-46b4-ae17-d7c22d322b16 -i td/osd-scrub-repair/0/new.json 2024-09-13T15:41:28.891 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:41:28.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-13T15:41:28.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAoXeRmApghJRAAN6vT5798KykrASFBjazlwA== --osd-uuid 97386219-fa2e-46b4-ae17-d7c22d322b16 2024-09-13T15:41:28.939 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:41:28.937+0000 7f8484c537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:41:28.976 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:41:28.973+0000 7f8484c537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:41:28.977 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:41:28.973+0000 7f8484c537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:41:28.978 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:41:28.973+0000 7f8484c537c0 -1 bdev(0x55b1cfd73c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:41:28.978 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:41:28.973+0000 7f8484c537c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-13T15:41:37.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-13T15:41:37.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:41:37.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:41:37.123 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:41:37.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:41:37.531 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:41:37.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:41:37.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:41:37.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:41:37.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:41:37.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:41:37.568 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:41:37.561+0000 7f29cdf057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:41:37.607 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:41:37.605+0000 7f29cdf057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:41:37.609 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:41:37.605+0000 7f29cdf057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:41:37.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:41:37.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:41:37.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:41:37.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:41:37.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:41:37.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:37.849 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:41:37.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:41:37.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:37.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:38.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:38.857 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:41:38.853+0000 7f29cdf057c0 -1 Falling back to public interface 2024-09-13T15:41:39.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:39.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:39.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:41:39.177 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:41:39.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:39.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:39.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:40.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:40.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:40.513 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:41:40.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:41:40.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:40.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:40.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:41.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:41.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:41.855 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:41:41.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:41:41.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:41.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:41.896 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:41:41.893+0000 7f29cdf057c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:41:42.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:43.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:43.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:43.308 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:41:43.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:41:43.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:43.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:43.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:44.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:44.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:44.763 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:41:44.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:41:44.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:44.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:45.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:46.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:46.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:46.324 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:41:46.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:41:46.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:46.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:46.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:47.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:47.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:47.775 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:41:47.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:41:47.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:47.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:48.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:49.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:49.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:49.214 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:41:49.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:41:49.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:49.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:49.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:50.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:50.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:50.550 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:41:50.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:41:50.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:50.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:51.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:52.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:52.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:52.043 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:41:52.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:41:52.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:52.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:52.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:53.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:53.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:53.371 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:41:53.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:41:53.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:53.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:53.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:54.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:54.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:54.698 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:41:54.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:41:54.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:54.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:55.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:56.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:56.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:56.198 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:41:56.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:41:56.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:56.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:56.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:57.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:57.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:57.525 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:41:57.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:41:57.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:57.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:57.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:41:58.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:41:58.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:41:58.974 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:41:58.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:41:58.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:41:58.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:41:59.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:00.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:00.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:00.293 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:42:00.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:42:00.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:00.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:42:00.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:01.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:01.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:01.630 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:42:01.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:42:01.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:01.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:42:02.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:03.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:03.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:03.401 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:42:03.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:42:03.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:03.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:42:04.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:05.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:05.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:05.492 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:42:05.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:42:05.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:05.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:42:06.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:07.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:07.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:07.499 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:42:07.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:42:07.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:07.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:42:08.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:09.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:09.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:09.230 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:42:09.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:42:09.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:09.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:42:09.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:10.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:10.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:10.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:42:10.912 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:42:10.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:10.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:42:12.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:13.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:13.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:13.053 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:42:13.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:42:13.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:13.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:42:13.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:14.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:14.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:14.476 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:42:14.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:42:14.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:14.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:42:14.810 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/90979981,v1:127.0.0.1:6803/90979981] [v2:127.0.0.1:6804/90979981,v1:127.0.0.1:6805/90979981] exists,up 97386219-fa2e-46b4-ae17-d7c22d322b16 2024-09-13T15:42:14.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:42:14.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:42:14.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:42:14.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-13T15:42:14.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2024-09-13T15:42:14.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:42:14.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:42:14.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T15:42:14.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:42:14.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-13T15:42:14.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:42:14.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:42:14.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:42:14.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:42:14.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-13T15:42:14.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-13T15:42:14.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:42:14.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:42:14.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:42:14.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:42:14.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:42:14.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:42:14.815 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:42:14.815 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:42:14.815 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:42:14.816 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:42:14.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:42:14.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:42:14.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:42:14.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:42:14.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:42:14.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:42:14.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:42:14.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:42:14.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:42:14.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:42:14.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:42:14.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:42:14.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-13T15:42:14.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:42:14.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ff14ad50-8a2f-42f4-abdc-cdbc47795251 2024-09-13T15:42:14.818 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 ff14ad50-8a2f-42f4-abdc-cdbc47795251 2024-09-13T15:42:14.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ff14ad50-8a2f-42f4-abdc-cdbc47795251' 2024-09-13T15:42:14.819 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:42:14.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBWXeRmzO4LMhAA/BT9RJsdbgJhlIjtFWAx3A== 2024-09-13T15:42:14.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBWXeRmzO4LMhAA/BT9RJsdbgJhlIjtFWAx3A=="}' 2024-09-13T15:42:14.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ff14ad50-8a2f-42f4-abdc-cdbc47795251 -i td/osd-scrub-repair/1/new.json 2024-09-13T15:42:15.443 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:42:15.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-13T15:42:15.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBWXeRmzO4LMhAA/BT9RJsdbgJhlIjtFWAx3A== --osd-uuid ff14ad50-8a2f-42f4-abdc-cdbc47795251 2024-09-13T15:42:15.503 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:42:15.504+0000 7fbb8c21b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:42:15.578 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:42:15.576+0000 7fbb8c21b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:42:15.580 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:42:15.580+0000 7fbb8c21b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:42:15.581 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:42:15.580+0000 7fbb8c21b7c0 -1 bdev(0x5642672c5c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-13T15:42:15.581 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:42:15.580+0000 7fbb8c21b7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-13T15:42:24.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-13T15:42:24.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:42:24.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T15:42:24.283 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T15:42:24.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:42:24.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T15:42:24.686 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:42:24.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:42:24.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:42:24.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:42:24.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:42:24.718 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:42:24.716+0000 7fb4132d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:42:24.766 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:42:24.764+0000 7fb4132d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:42:24.768 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:42:24.768+0000 7fb4132d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:42:24.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T15:42:24.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:42:24.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:42:24.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:42:24.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:42:24.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:24.991 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:42:24.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:42:24.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:24.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:25.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:25.797 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:42:25.796+0000 7fb4132d37c0 -1 Falling back to public interface 2024-09-13T15:42:26.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:26.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:26.364 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:42:26.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:42:26.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:26.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:26.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:27.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:27.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:27.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:42:27.699 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:42:27.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:27.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:28.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:29.040 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:42:29.040+0000 7fb4132d37c0 -1 osd.1 0 log_to_monitors true 2024-09-13T15:42:29.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:29.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:29.043 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:42:29.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:42:29.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:29.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:29.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:30.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:30.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:30.371 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:42:30.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:42:30.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:30.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:30.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:31.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:31.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:31.706 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:42:31.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:42:31.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:31.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:32.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:33.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:33.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:33.179 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:42:33.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:42:33.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:33.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:33.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:34.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:34.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:34.518 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:42:34.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:42:34.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:34.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:34.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:35.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:35.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:35.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:42:35.892 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:42:35.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:35.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:36.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:37.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:37.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:37.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:42:37.747 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:42:37.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:37.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:38.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:39.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:39.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:39.219 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:42:39.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:42:39.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:39.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:39.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:40.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:40.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:40.555 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:42:40.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:42:40.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:40.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:40.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:41.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:41.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:41.892 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:42:41.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:42:41.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:41.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:42.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:43.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:43.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:43.383 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:42:43.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:42:43.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:43.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:43.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:44.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:44.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:44.774 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:42:44.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:42:44.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:44.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:45.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:46.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:46.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:46.176 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:42:46.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:42:46.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:46.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:46.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:47.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:47.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:47.540 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:42:47.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:42:47.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:47.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:47.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:48.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:48.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:48.874 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:42:48.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:42:48.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:48.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:49.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:50.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:50.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:50.496 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:42:50.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:42:50.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:50.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:50.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:51.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:51.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:51.899 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:42:51.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:42:51.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:51.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:52.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:53.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:53.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:53.640 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:42:53.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:42:53.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:53.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:54.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:55.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:55.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:55.443 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:42:55.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:42:55.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:55.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:55.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:56.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:56.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:56.889 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:42:56.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:42:56.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:56.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:58.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:42:59.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:42:59.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:42:59.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:42:59.004 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:42:59.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:42:59.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:42:59.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:00.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:00.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:00.461 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:43:00.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:43:00.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:00.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:43:01.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:02.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:02.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:02.018 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:43:02.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:43:02.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:02.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:43:02.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:03.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:03.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:03.385 INFO:tasks.workunit.client.0.smithi112.stdout:26 2024-09-13T15:43:03.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 26 2024-09-13T15:43:03.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:03.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:43:03.714 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4021979661,v1:127.0.0.1:6811/4021979661] [v2:127.0.0.1:6812/4021979661,v1:127.0.0.1:6813/4021979661] exists,up ff14ad50-8a2f-42f4-abdc-cdbc47795251 2024-09-13T15:43:03.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:43:03.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:43:03.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:43:03.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-13T15:43:03.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2024-09-13T15:43:03.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:43:03.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:43:03.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T15:43:03.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:43:03.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:43:03.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:43:03.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:43:03.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:43:03.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:43:03.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T15:43:03.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T15:43:03.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:43:03.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:43:03.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:43:03.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:43:03.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:43:03.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:43:03.719 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:43:03.719 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:43:03.719 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:43:03.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:43:03.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:43:03.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:43:03.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:43:03.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:43:03.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:43:03.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:43:03.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:43:03.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:43:03.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:43:03.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:43:03.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:43:03.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:43:03.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T15:43:03.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:43:03.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5ebab4b3-f99e-4bde-8bde-6974d25736c2 2024-09-13T15:43:03.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 5ebab4b3-f99e-4bde-8bde-6974d25736c2' 2024-09-13T15:43:03.723 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 5ebab4b3-f99e-4bde-8bde-6974d25736c2 2024-09-13T15:43:03.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:43:03.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCHXeRmY6UMLBAAT/g0uyjaaevB1pZZX2vY6A== 2024-09-13T15:43:03.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCHXeRmY6UMLBAAT/g0uyjaaevB1pZZX2vY6A=="}' 2024-09-13T15:43:03.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5ebab4b3-f99e-4bde-8bde-6974d25736c2 -i td/osd-scrub-repair/2/new.json 2024-09-13T15:43:04.186 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:43:04.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-13T15:43:04.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHXeRmY6UMLBAAT/g0uyjaaevB1pZZX2vY6A== --osd-uuid 5ebab4b3-f99e-4bde-8bde-6974d25736c2 2024-09-13T15:43:04.243 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:43:04.239+0000 7ff99ddb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:43:04.321 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:43:04.319+0000 7ff99ddb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:43:04.323 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:43:04.323+0000 7ff99ddb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:43:04.323 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:43:04.323+0000 7ff99ddb97c0 -1 bdev(0x5580ba7e0800 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-13T15:43:04.324 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:43:04.323+0000 7ff99ddb97c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-13T15:43:13.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-13T15:43:13.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:43:13.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T15:43:13.251 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T15:43:13.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:43:13.727 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T15:43:13.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T15:43:13.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:43:13.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:43:13.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:43:13.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:43:13.761 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:43:13.758+0000 7f3b8ab187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:43:13.788 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:43:13.786+0000 7f3b8ab187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:43:13.789 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:43:13.786+0000 7f3b8ab187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:43:14.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T15:43:14.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:43:14.039 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:43:14.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:43:14.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:43:14.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:43:14.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:14.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:43:14.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:14.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:14.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:15.281 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:43:15.278+0000 7f3b8ab187c0 -1 Falling back to public interface 2024-09-13T15:43:15.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:15.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:15.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:43:15.380 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:43:15.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:15.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:15.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:16.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:16.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:16.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:43:16.722 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:43:16.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:16.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:17.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:18.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:18.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:18.048 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:43:18.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:43:18.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:18.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:18.360 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:43:18.358+0000 7f3b8ab187c0 -1 osd.2 0 log_to_monitors true 2024-09-13T15:43:18.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:19.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:19.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:19.388 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:43:19.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:43:19.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:19.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:19.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:20.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:20.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:20.983 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:43:20.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:43:20.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:20.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:21.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:22.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:22.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:22.320 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:43:22.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:43:22.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:22.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:22.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:23.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:23.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:23.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:43:23.731 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:43:23.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:23.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:24.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:25.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:25.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:25.204 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:43:25.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:43:25.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:25.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:25.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:26.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:26.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:26.651 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:43:26.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:43:26.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:26.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:26.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:27.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:27.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:27.964 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:43:27.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:43:27.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:27.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:28.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:29.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:29.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:29.358 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:43:29.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:43:29.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:29.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:29.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:30.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:30.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:30.718 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:43:30.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:43:30.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:30.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:31.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:32.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:32.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:32.044 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:43:32.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:43:32.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:32.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:32.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:33.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:33.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:33.511 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:43:33.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:43:33.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:33.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:33.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:34.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:34.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:34.953 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:43:34.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:43:34.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:34.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:35.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:36.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:36.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:36.286 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:43:36.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:43:36.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:36.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:36.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:37.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:37.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:37.748 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:43:37.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:43:37.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:37.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:38.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:39.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:39.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:39.174 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:43:39.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:43:39.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:39.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:40.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:41.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:41.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:41.119 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:43:41.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:43:41.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:41.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:42.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:43.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:43.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:43.138 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:43:43.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:43:43.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:43.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:44.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:45.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:45.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:45.151 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:43:45.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:43:45.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:45.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:46.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:47.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:47.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:47.373 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:43:47.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:43:47.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:47.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:48.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:49.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:49.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:49.456 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:43:49.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:43:49.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:49.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:49.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:50.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:50.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:50.784 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:43:50.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:43:50.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:50.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:51.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:43:52.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:43:52.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:43:52.118 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:43:52.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:43:52.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:43:52.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:43:52.440 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1704023335,v1:127.0.0.1:6819/1704023335] [v2:127.0.0.1:6820/1704023335,v1:127.0.0.1:6821/1704023335] exists,up 5ebab4b3-f99e-4bde-8bde-6974d25736c2 2024-09-13T15:43:52.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:43:52.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:43:52.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:43:52.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-09-13T15:43:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2024-09-13T15:43:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-13T15:43:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:43:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-13T15:43:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:43:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T15:43:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:43:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:43:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:43:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:43:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T15:43:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T15:43:52.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:43:52.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:43:52.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:43:52.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:43:52.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:43:52.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:43:52.444 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:43:52.444 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:43:52.444 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:43:52.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:43:52.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:43:52.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:43:52.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:43:52.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:43:52.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:43:52.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:43:52.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:43:52.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:43:52.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:43:52.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:43:52.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:43:52.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:43:52.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T15:43:52.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:43:52.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e4c6ac86-74e0-43c9-99ec-f965fd8333e0 2024-09-13T15:43:52.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 e4c6ac86-74e0-43c9-99ec-f965fd8333e0' 2024-09-13T15:43:52.447 INFO:tasks.workunit.client.0.smithi112.stdout:add osd3 e4c6ac86-74e0-43c9-99ec-f965fd8333e0 2024-09-13T15:43:52.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:43:52.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC4XeRmK0YSHBAArYxFzIKim7wndH4TbJtEig== 2024-09-13T15:43:52.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC4XeRmK0YSHBAArYxFzIKim7wndH4TbJtEig=="}' 2024-09-13T15:43:52.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e4c6ac86-74e0-43c9-99ec-f965fd8333e0 -i td/osd-scrub-repair/3/new.json 2024-09-13T15:43:52.983 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:43:53.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-13T15:43:53.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4XeRmK0YSHBAArYxFzIKim7wndH4TbJtEig== --osd-uuid e4c6ac86-74e0-43c9-99ec-f965fd8333e0 2024-09-13T15:43:53.044 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:43:53.041+0000 7f8889a287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:43:53.119 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:43:53.117+0000 7f8889a287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:43:53.120 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:43:53.117+0000 7f8889a287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:43:53.121 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:43:53.117+0000 7f8889a287c0 -1 bdev(0x5565f5a33c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-13T15:43:53.121 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:43:53.117+0000 7f8889a287c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-13T15:44:02.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-13T15:44:02.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:44:02.340 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd3 key to auth repository 2024-09-13T15:44:02.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-13T15:44:02.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:44:02.732 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.3 2024-09-13T15:44:02.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-13T15:44:02.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:44:02.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:44:02.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:44:02.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:44:02.765 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:44:02.761+0000 7fdd79e5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:44:02.767 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:44:02.765+0000 7fdd79e5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:44:02.769 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:44:02.765+0000 7fdd79e5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:44:03.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-13T15:44:03.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:44:03.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T15:44:03.055 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:44:03.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:44:03.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:44:03.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:03.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:44:03.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:03.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:03.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:03.885 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:44:03.881+0000 7fdd79e5a7c0 -1 Falling back to public interface 2024-09-13T15:44:04.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:04.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:04.428 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:44:04.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:44:04.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:04.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:04.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:05.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:05.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:05.761 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:44:05.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:44:05.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:05.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:06.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:06.964 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:44:06.961+0000 7fdd79e5a7c0 -1 osd.3 0 log_to_monitors true 2024-09-13T15:44:07.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:07.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:07.101 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:44:07.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:44:07.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:07.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:07.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:08.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:08.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:08.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:44:08.679 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:44:08.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:08.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:09.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:10.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:10.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:10.064 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:44:10.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:44:10.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:10.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:10.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:11.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:11.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:11.387 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:44:11.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:44:11.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:11.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:11.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:12.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:12.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:12.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:44:12.802 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:44:12.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:12.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:13.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:14.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:14.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:14.141 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:44:14.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:44:14.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:14.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:14.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:15.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:15.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:15.467 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:44:15.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:44:15.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:15.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:15.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:16.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:16.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:16.794 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:44:16.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:44:16.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:16.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:17.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:18.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:18.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:18.262 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:44:18.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:44:18.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:18.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:18.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:19.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:19.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:19.592 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:44:19.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:44:19.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:19.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:20.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:21.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:21.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:21.083 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:44:21.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:44:21.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:21.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:21.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:22.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:22.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:22.457 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:44:22.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:44:22.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:22.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:22.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:23.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:23.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:23.804 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:44:23.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:44:23.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:23.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:24.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:25.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:25.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:25.348 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:44:25.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:44:25.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:25.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:25.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:26.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:26.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:26.684 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:44:26.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:44:26.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:26.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:27.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:28.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:28.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:28.065 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:44:28.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:44:28.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:28.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:28.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:29.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:29.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:29.901 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:44:29.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:44:29.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:29.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:30.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:31.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:31.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:31.661 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:44:31.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:44:31.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:31.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:32.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:33.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:33.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:33.715 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:44:33.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:44:33.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:33.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:34.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:35.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:35.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:35.422 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:44:35.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:44:35.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:35.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:36.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:37.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:37.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:37.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:44:37.172 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:44:37.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:37.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:37.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:38.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:38.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:38.886 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:44:38.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:44:38.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:38.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:39.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:44:40.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:44:40.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:44:40.277 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:44:40.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:44:40.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:44:40.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:44:40.602 INFO:tasks.workunit.client.0.smithi112.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1713030697,v1:127.0.0.1:6827/1713030697] [v2:127.0.0.1:6828/1713030697,v1:127.0.0.1:6829/1713030697] exists,up e4c6ac86-74e0-43c9-99ec-f965fd8333e0 2024-09-13T15:44:40.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:44:40.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:44:40.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:44:40.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: create_ec_pool ecpool true k=2 m=2 2024-09-13T15:44:40.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-13T15:44:40.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-13T15:44:40.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-13T15:44:40.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-13T15:44:40.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-13T15:44:41.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-13T15:44:41.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-13T15:44:42.223 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'ecpool' created 2024-09-13T15:44:42.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:44:43.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-13T15:44:43.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-13T15:44:44.567 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 1 allow_ec_overwrites to true 2024-09-13T15:44:44.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-13T15:44:44.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:44:44.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:44:44.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:44:44.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:44:44.595 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:44:44.595 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:44:44.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:44:44.596 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:44:44.596 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:44:44.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:44:44.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:44:44.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:44:44.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:44:44.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:44:44.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:44:45.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:44:45.181 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:44:45.181 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:44:45.181 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T15:44:45.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:44:45.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:44:45.181 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:44:45.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836523 2024-09-13T15:44:45.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836523 2024-09-13T15:44:45.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836523' 2024-09-13T15:44:45.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:44:45.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:44:45.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672984 2024-09-13T15:44:45.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672984 2024-09-13T15:44:45.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836523 1-42949672984' 2024-09-13T15:44:45.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:44:45.559 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:44:45.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509457 2024-09-13T15:44:45.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509457 2024-09-13T15:44:45.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836523 1-42949672984 2-64424509457' 2024-09-13T15:44:45.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:44:45.746 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T15:44:45.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345929 2024-09-13T15:44:45.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345929 2024-09-13T15:44:45.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836523 1-42949672984 2-64424509457 3-85899345929' 2024-09-13T15:44:45.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:44:45.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836523 2024-09-13T15:44:45.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:44:45.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:44:45.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836523 2024-09-13T15:44:45.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:44:45.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836523 2024-09-13T15:44:45.900 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836523 2024-09-13T15:44:45.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-09-13T15:44:45.900 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:44:46.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836522 -lt 21474836523 2024-09-13T15:44:46.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:44:47.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:44:47.208 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:44:47.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836523 -lt 21474836523 2024-09-13T15:44:47.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:44:47.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672984 2024-09-13T15:44:47.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:44:47.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:44:47.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672984 2024-09-13T15:44:47.542 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:44:47.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672984 2024-09-13T15:44:47.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-13T15:44:47.543 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672984 2024-09-13T15:44:47.544 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:44:47.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672984 -lt 42949672984 2024-09-13T15:44:47.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:44:47.855 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509457 2024-09-13T15:44:47.856 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:44:47.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:44:47.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509457 2024-09-13T15:44:47.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:44:47.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509457 2024-09-13T15:44:47.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-13T15:44:47.860 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509457 2024-09-13T15:44:47.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:44:48.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509457 2024-09-13T15:44:48.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:44:48.208 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345929 2024-09-13T15:44:48.208 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:44:48.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T15:44:48.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345929 2024-09-13T15:44:48.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:44:48.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345929 2024-09-13T15:44:48.211 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.3 seq 85899345929 2024-09-13T15:44:48.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2024-09-13T15:44:48.211 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T15:44:48.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345929 -lt 85899345929 2024-09-13T15:44:48.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:44:48.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:44:48.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:44:48.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:44:48.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:44:48.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:44:48.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:44:48.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:44:48.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:44:48.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:44:48.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:44:49.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:44:49.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:44:49.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:44:49.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:44:49.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:44:49.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:44:49.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:44:49.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-13T15:44:49.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-13T15:44:49.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-13T15:44:49.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-13T15:44:49.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-13T15:44:49.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-13T15:44:49.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-13T15:44:49.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-13T15:44:50.192 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:44:50.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-13T15:44:50.803 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:44:50.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-13T15:44:50.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-13T15:44:50.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-13T15:44:51.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: get_primary ecpool SOMETHING 2024-09-13T15:44:51.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-13T15:44:51.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T15:44:51.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:44:51.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-13T15:44:51.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: local primary=1 2024-09-13T15:44:51.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: get_osds ecpool SOMETHING 2024-09-13T15:44:51.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: sed -e s/1// 2024-09-13T15:44:51.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-13T15:44:51.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-13T15:44:51.664 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-13T15:44:51.665 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-13T15:44:51.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-13T15:44:51.983 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T15:44:51.983 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T15:44:51.983 INFO:tasks.workunit.client.0.smithi112.stderr:3' 2024-09-13T15:44:51.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 3 2024-09-13T15:44:51.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: osds=('0' '2' '3') 2024-09-13T15:44:51.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: local -a osds 2024-09-13T15:44:51.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: local not_primary_first=0 2024-09-13T15:44:51.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: local not_primary_second=2 2024-09-13T15:44:51.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:871: unfound_erasure_coded: local not_primary_third=3 2024-09-13T15:44:51.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: pids= 2024-09-13T15:44:51.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T15:44:51.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T15:44:51.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T15:44:51.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 466552"' 2024-09-13T15:44:51.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 466552' 2024-09-13T15:44:51.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T15:44:51.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T15:44:51.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T15:44:51.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 466553"' 2024-09-13T15:44:51.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 466553' 2024-09-13T15:44:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T15:44:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-13T15:44:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-13T15:44:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T15:44:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/466555: /' 2024-09-13T15:44:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 466556"' 2024-09-13T15:44:51.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 466556' 2024-09-13T15:44:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: wait_background pids 2024-09-13T15:44:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 466552 466553 466556' 2024-09-13T15:44:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-13T15:44:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T15:44:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 466552 2024-09-13T15:44:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T15:44:51.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/466559: /' 2024-09-13T15:44:51.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T15:44:51.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/466561: /' 2024-09-13T15:44:57.053 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:44:57.053 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:44:57.053 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-13T15:44:57.053 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:44:57.054 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T15:44:57.054 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:44:57.054 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:44:57.054 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-13T15:44:57.054 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:44:57.054 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-13T15:44:57.054 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:44:57.054 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:44:57.054 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:44:57.054 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:44:57.055 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:44:57.055 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:44:57.055 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-13T15:44:57.055 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:44:57.055 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:44:57.055 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-13T15:44:57.055 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:44:57.055 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:44:57.055 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-13T15:44:57.055 INFO:tasks.workunit.client.0.smithi112.stderr:466559: remove 2#1:eb822e21:::SOMETHING:head# 2024-09-13T15:44:57.055 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-13T15:44:57.055 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:44:57.055 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:44:57.055 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T15:44:57.056 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:44:57.056 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-13T15:44:57.056 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:44:57.056 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:44:57.059 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:44:57.059 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:44:57.059 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-13T15:44:57.059 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-13T15:44:57.059 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:44:57.059 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:44:57.059 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:44:57.059 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:44:57.060 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:44:57.060 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:44:57.060 INFO:tasks.workunit.client.0.smithi112.stderr:466559: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:44:57.060 INFO:tasks.workunit.client.0.smithi112.stderr:466559: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:44:57.060 INFO:tasks.workunit.client.0.smithi112.stderr:466559: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:44:57.060 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:44:57.060 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:44:57.060 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:44:57.060 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:44:57.060 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:44:57.060 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:44:57.060 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:44:57.061 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:44:57.061 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:44:57.061 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:44:57.061 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:44:57.061 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-13T15:44:57.061 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T15:44:57.061 INFO:tasks.workunit.client.0.smithi112.stderr:466559: start osd.2 2024-09-13T15:44:57.061 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:44:57.732 INFO:tasks.workunit.client.0.smithi112.stderr:4466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:44:57.732 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:44:57.733 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-13T15:44:57.733 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:44:57.733 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T15:44:57.733 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:44:57.733 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:44:57.733 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-13T15:44:57.733 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:44:57.733 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-13T15:44:57.733 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:44:57.733 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:44:57.733 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:44:57.733 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:44:57.733 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:44:57.733 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:44:57.734 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-13T15:44:57.734 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:44:57.734 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:44:57.734 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-13T15:44:57.734 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:44:57.734 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:44:57.734 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-13T15:44:57.734 INFO:tasks.workunit.client.0.smithi112.stderr:466555: remove 1#1:eb822e21:::SOMETHING:head# 2024-09-13T15:44:57.734 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-13T15:44:57.734 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:44:57.734 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:44:57.734 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:44:57.734 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:44:57.734 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-13T15:44:57.734 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:44:57.735 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:44:57.736 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:44:57.736 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:44:57.736 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-13T15:44:57.737 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-13T15:44:57.737 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:44:57.737 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:44:57.737 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:44:57.737 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:44:57.737 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:44:57.737 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:44:57.737 INFO:tasks.workunit.client.0.smithi112.stderr:466555: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:44:57.737 INFO:tasks.workunit.client.0.smithi112.stderr:466555: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:44:57.737 INFO:tasks.workunit.client.0.smithi112.stderr:466555: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:44:57.737 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:44:57.737 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:44:57.737 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:44:57.738 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:44:57.738 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:44:57.738 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:44:57.738 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:44:57.738 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:44:57.738 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:44:57.738 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:44:57.738 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:44:57.738 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-13T15:44:57.738 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:44:57.738 INFO:tasks.workunit.client.0.smithi112.stderr:466555: start osd.0 2024-09-13T15:44:57.738 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:44:57.802 INFO:tasks.workunit.client.0.smithi112.stderr:4466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-13T15:44:57.802 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T15:44:57.802 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-13T15:44:57.802 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T15:44:57.802 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T15:44:57.804 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-13T15:44:57.804 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T15:44:57.804 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-13T15:44:57.804 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T15:44:57.804 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-13T15:44:57.804 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:44:57.804 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:44:57.804 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:44:57.804 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:44:57.804 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:44:57.804 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:44:57.805 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-13T15:44:57.805 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-13T15:44:57.805 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T15:44:57.805 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-13T15:44:57.805 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T15:44:57.805 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-13T15:44:57.805 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-13T15:44:57.805 INFO:tasks.workunit.client.0.smithi112.stderr:466561: remove 3#1:eb822e21:::SOMETHING:head# 2024-09-13T15:44:57.805 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-13T15:44:57.805 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-13T15:44:57.806 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:44:57.806 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-13T15:44:57.806 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:44:57.806 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-13T15:44:57.806 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-13T15:44:57.806 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:44:57.806 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:44:57.806 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:44:57.807 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-13T15:44:57.807 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-13T15:44:57.807 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:44:57.807 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:44:57.807 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-13T15:44:57.807 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:44:57.807 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:44:57.807 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:44:57.807 INFO:tasks.workunit.client.0.smithi112.stderr:466561: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:44:57.807 INFO:tasks.workunit.client.0.smithi112.stderr:466561: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:44:57.807 INFO:tasks.workunit.client.0.smithi112.stderr:466561: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:44:57.807 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:44:57.807 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' 2024-09-13T15:44:57.808 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:44:57.808 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-13T15:44:57.808 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-13T15:44:57.808 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:44:57.808 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:44:57.808 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:44:57.808 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:44:57.808 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:44:57.808 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:44:57.808 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-13T15:44:57.808 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-13T15:44:57.808 INFO:tasks.workunit.client.0.smithi112.stderr:466561: start osd.3 2024-09-13T15:44:57.808 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=0caebd5d-b552-471f-951a-4f9b301221ff --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.63282/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:45:02.296 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-13T15:45:02.296 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T15:45:02.297 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:45:02.297 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:45:02.297 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:45:02.297 INFO:tasks.workunit.client.0.smithi112.stderr:466559: 2024-09-13T15:44:57.092+0000 7f2f028b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:45:02.297 INFO:tasks.workunit.client.0.smithi112.stderr:466559: 2024-09-13T15:44:57.092+0000 7f2f028b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:45:02.297 INFO:tasks.workunit.client.0.smithi112.stderr:466559: 2024-09-13T15:44:57.096+0000 7f2f028b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:45:02.297 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T15:45:02.297 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:45:02.297 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:45:02.297 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:45:02.297 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:45:02.297 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:02.297 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:45:02.298 INFO:tasks.workunit.client.0.smithi112.stderr:466559: 0 2024-09-13T15:45:02.298 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:02.298 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:45:02.298 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:02.298 INFO:tasks.workunit.client.0.smithi112.stderr:466559: 2024-09-13T15:44:58.664+0000 7f2f028b27c0 -1 Falling back to public interface 2024-09-13T15:45:02.298 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:02.298 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:02.298 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:45:02.298 INFO:tasks.workunit.client.0.smithi112.stderr:466559: 1 2024-09-13T15:45:02.298 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:02.298 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:45:02.298 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:02.299 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:02.299 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:02.299 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:45:02.299 INFO:tasks.workunit.client.0.smithi112.stderr:466559: 2 2024-09-13T15:45:02.299 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:02.299 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:45:02.299 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:02.299 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:02.299 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:02.299 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:45:02.299 INFO:tasks.workunit.client.0.smithi112.stderr:466559: 3 2024-09-13T15:45:02.299 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:02.299 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:45:02.300 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:03.479 INFO:tasks.workunit.client.0.smithi112.stderr:466559: 2024-09-13T15:45:02.295+0000 7f2f028b27c0 -66555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-13T15:45:03.479 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:45:03.479 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:45:03.479 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:45:03.479 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:45:03.479 INFO:tasks.workunit.client.0.smithi112.stderr:466555: 2024-09-13T15:44:57.764+0000 7fa34ef2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:45:03.479 INFO:tasks.workunit.client.0.smithi112.stderr:466555: 2024-09-13T15:44:57.792+0000 7fa34ef2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:45:03.479 INFO:tasks.workunit.client.0.smithi112.stderr:466555: 2024-09-13T15:44:57.796+0000 7fa34ef2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:45:03.479 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:45:03.479 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:45:03.480 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:45:03.480 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:45:03.480 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:45:03.480 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:03.480 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:45:03.480 INFO:tasks.workunit.client.0.smithi112.stderr:466555: 0 2024-09-13T15:45:03.480 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:03.480 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:03.480 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:03.480 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:03.480 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:03.480 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:45:03.480 INFO:tasks.workunit.client.0.smithi112.stderr:466555: 1 2024-09-13T15:45:03.481 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:03.481 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:03.481 INFO:tasks.workunit.client.0.smithi112.stderr:466555: 2024-09-13T15:44:59.648+0000 7fa34ef2b7c0 -1 Falling back to public interface 2024-09-13T15:45:03.481 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:03.481 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:03.481 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:03.481 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:45:03.481 INFO:tasks.workunit.client.0.smithi112.stderr:466555: 2 2024-09-13T15:45:03.481 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:03.481 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:03.481 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:03.481 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:03.481 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:03.481 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:45:03.481 INFO:tasks.workunit.client.0.smithi112.stderr:466555: 3 2024-09-13T15:45:03.482 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:03.482 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:03.482 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:03.492 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/sta66561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-13T15:45:03.492 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-13T15:45:03.492 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:45:03.492 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:45:03.493 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:45:03.493 INFO:tasks.workunit.client.0.smithi112.stderr:466561: 2024-09-13T15:44:57.832+0000 7f523029d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:45:03.493 INFO:tasks.workunit.client.0.smithi112.stderr:466561: 2024-09-13T15:44:57.832+0000 7f523029d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:45:03.493 INFO:tasks.workunit.client.0.smithi112.stderr:466561: 2024-09-13T15:44:57.836+0000 7f523029d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:45:03.493 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-13T15:45:03.493 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:45:03.493 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-13T15:45:03.493 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:45:03.493 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:45:03.493 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:03.493 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:45:03.493 INFO:tasks.workunit.client.0.smithi112.stderr:466561: 0 2024-09-13T15:45:03.493 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:03.493 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:45:03.494 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:03.494 INFO:tasks.workunit.client.0.smithi112.stderr:466561: 2024-09-13T15:44:59.156+0000 7f523029d7c0 -1 Falling back to public interface 2024-09-13T15:45:03.494 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:03.494 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:03.494 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:45:03.494 INFO:tasks.workunit.client.0.smithi112.stderr:466561: 1 2024-09-13T15:45:03.494 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:03.494 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:45:03.494 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:03.494 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:03.494 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:03.494 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:45:03.495 INFO:tasks.workunit.client.0.smithi112.stderr:466561: 2 2024-09-13T15:45:03.495 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:03.495 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:45:03.495 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:03.495 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:03.495 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:03.495 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:45:03.495 INFO:tasks.workunit.client.0.smithi112.stderr:466561: 3 2024-09-13T15:45:03.495 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:03.495 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:45:03.495 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/sta1 osd.2 29 log_to_monitors true 2024-09-13T15:45:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:45:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:466559: 4 2024-09-13T15:45:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:45:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:466559: osd.2 up in weight 1 up_from 33 up_thru 0 down_at 30 last_clean_interval [15,29) [v2:127.0.0.1:6802/2414914177,v1:127.0.0.1:6803/2414914177] [v2:127.0.0.1:6804/2414914177,v1:127.0.0.1:6805/2414914177] exists,up 5ebab4b3-f99e-4bde-8bde-6974d25736c2 2024-09-13T15:45:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:45:04.388 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:45:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:45:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:45:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:45:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:45:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:45:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:45:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:466559: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:45:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:466559: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:45:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:45:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:45:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:45:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:45:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:45:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:45:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:45:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:45:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:45:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:45:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:466559: 1 2024-09-13T15:45:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:466559: 2 2024-09-13T15:45:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:466559: 3' 2024-09-13T15:45:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:45:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:45:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:45:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:466559: Error ENXIO: problem getting command descriptions from osd.0 2024-09-13T15:45:04.391 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-13T15:45:04.391 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-13T15:45:04.391 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-13T15:45:04.391 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:45:05.929 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clondalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:05.929 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:05.929 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:45:05.930 INFO:tasks.workunit.client.0.smithi112.stderr:466555: 4 2024-09-13T15:45:05.930 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:05.930 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:05.930 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:05.930 INFO:tasks.workunit.client.0.smithi112.stderr:466555: 2024-09-13T15:45:04.075+0000 7fa34ef2b7c0 -1 osd.0 29 log_to_monitors true 2024-09-13T15:45:05.930 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:05.930 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:05.930 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:45:05.930 INFO:tasks.workunit.client.0.smithi112.stderr:466555: 5 2024-09-13T15:45:05.930 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:05.930 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:05.930 INFO:tasks.workunit.client.0.smithi112.stderr:466555: osd.0 up in weight 1 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6826/1536760333,v1:127.0.0.1:6827/1536760333] [v2:127.0.0.1:6828/1536760333,v1:127.0.0.1:6829/1536760333] exists,up 97386219-fa2e-46b4-ae17-d7c22d322b16 2024-09-13T15:45:05.930 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:45:05.931 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:45:05.931 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:45:05.931 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:45:05.931 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:45:05.931 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:45:05.931 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:45:05.931 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:45:05.931 INFO:tasks.workunit.client.0.smithi112.stderr:466555: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:45:05.931 INFO:tasks.workunit.client.0.smithi112.stderr:466555: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:45:05.931 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:45:05.931 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:45:05.931 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:45:05.931 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:45:05.931 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:45:05.932 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:45:05.932 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:45:05.932 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:45:05.932 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:45:05.932 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:45:05.932 INFO:tasks.workunit.client.0.smithi112.stderr:466555: 1 2024-09-13T15:45:05.932 INFO:tasks.workunit.client.0.smithi112.stderr:466555: 2 2024-09-13T15:45:05.932 INFO:tasks.workunit.client.0.smithi112.stderr:466555: 3' 2024-09-13T15:45:05.932 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:45:05.932 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:45:06.141 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubundalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:06.142 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:06.142 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:45:06.142 INFO:tasks.workunit.client.0.smithi112.stderr:466561: 4 2024-09-13T15:45:06.142 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:06.142 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:45:06.142 INFO:tasks.workunit.client.0.smithi112.stderr:466561: 2024-09-13T15:45:03.507+0000 7f523029d7c0 -1 osd.3 29 log_to_monitors true 2024-09-13T15:45:06.142 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:06.142 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:06.142 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:06.142 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:45:06.142 INFO:tasks.workunit.client.0.smithi112.stderr:466561: 5 2024-09-13T15:45:06.143 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:06.143 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-13T15:45:06.143 INFO:tasks.workunit.client.0.smithi112.stderr:466561: osd.3 up in weight 1 up_from 35 up_thru 0 down_at 30 last_clean_interval [20,29) [v2:127.0.0.1:6818/1835396125,v1:127.0.0.1:6819/1835396125] [v2:127.0.0.1:6820/1835396125,v1:127.0.0.1:6821/1835396125] exists,up e4c6ac86-74e0-43c9-99ec-f965fd8333e0 2024-09-13T15:45:06.143 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:45:06.143 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:45:06.143 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:45:06.143 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T15:45:06.143 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:45:06.143 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:45:06.143 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:45:06.143 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:45:06.143 INFO:tasks.workunit.client.0.smithi112.stderr:466561: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:45:06.143 INFO:tasks.workunit.client.0.smithi112.stderr:466561: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:45:06.143 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:45:06.144 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:45:06.144 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:45:06.144 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:45:06.144 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:45:06.144 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:45:06.144 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:45:06.144 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:45:06.144 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:45:06.144 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:45:06.144 INFO:tasks.workunit.client.0.smithi112.stderr:466561: 1 2024-09-13T15:45:06.144 INFO:tasks.workunit.client.0.smithi112.stderr:466561: 2 2024-09-13T15:45:06.145 INFO:tasks.workunit.client.0.smithi112.stderr:466561: 3' 2024-09-13T15:45:06.145 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:45:06.145 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:45:06.337 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubne.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:45:06.337 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672989 2024-09-13T15:45:06.337 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672989 2024-09-13T15:45:06.337 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 1-42949672989' 2024-09-13T15:45:06.337 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:45:06.337 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:45:06.338 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920770 2024-09-13T15:45:06.338 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920770 2024-09-13T15:45:06.338 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 1-42949672989 2-141733920770' 2024-09-13T15:45:06.338 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:45:06.338 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T15:45:06.338 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-13T15:45:06.338 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-13T15:45:06.338 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 1-42949672989 2-141733920770 3-150323855362' 2024-09-13T15:45:06.338 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:45:06.338 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672989 2024-09-13T15:45:06.338 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:45:06.338 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:45:06.338 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672989 2024-09-13T15:45:06.338 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:45:06.339 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672989 2024-09-13T15:45:06.339 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-09-13T15:45:06.339 INFO:tasks.workunit.client.0.smithi112.stderr:466559: waiting osd.1 seq 42949672989 2024-09-13T15:45:06.339 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:45:06.339 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672989 -lt 42949672989 2024-09-13T15:45:06.339 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:45:06.339 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920770 2024-09-13T15:45:06.339 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:45:06.339 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:45:06.339 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920770 2024-09-13T15:45:06.339 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:45:06.339 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920770 2024-09-13T15:45:06.339 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920770' 2024-09-13T15:45:06.339 INFO:tasks.workunit.client.0.smithi112.stderr:466559: waiting osd.2 seq 141733920770 2024-09-13T15:45:06.340 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:45:08.225 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920770 -lt 141733920770 2024-09-13T15:45:08.225 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:45:08.225 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-150323855362 2024-09-13T15:45:08.225 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:45:08.225 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T15:45:08.225 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-150323855362 2024-09-13T15:45:08.226 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:45:08.226 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-13T15:45:08.226 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 150323855362' 2024-09-13T15:45:08.226 INFO:tasks.workunit.client.0.smithi112.stderr:466559: waiting osd.3 seq 150323855362 2024-09-13T15:45:08.226 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T15:45:08.226 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-13T15:45:08.226 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:45:08.226 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:45:08.226 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:45:08.226 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:45:08.226 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:45:08.226 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:45:08.226 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:45:08.226 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:45:08.227 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:45:08.227 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:45:08.227 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:45:08.227 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T15:45:08.227 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:45:08.227 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:45:08.227 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:45:08.227 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T15:45:08.227 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T15:45:08.227 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T15:45:08.227 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T15:45:08.227 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:45:08.228 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T15:45:08.228 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:45:08.228 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:45:08.524 INFO:tasks.workunit.client.0.smithi112.stderr:466559:ntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:45:08.524 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822658 2024-09-13T15:45:08.524 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822658 2024-09-13T15:45:08.524 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822658' 2024-09-13T15:45:08.524 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:45:08.524 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:45:08.524 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672990 2024-09-13T15:45:08.524 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672990 2024-09-13T15:45:08.524 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822658 1-42949672990' 2024-09-13T15:45:08.524 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:45:08.525 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:45:08.525 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920771 2024-09-13T15:45:08.525 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920771 2024-09-13T15:45:08.525 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822658 1-42949672990 2-141733920771' 2024-09-13T15:45:08.525 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:45:08.525 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T15:45:08.525 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855363 2024-09-13T15:45:08.525 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855363 2024-09-13T15:45:08.525 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822658 1-42949672990 2-141733920771 3-150323855363' 2024-09-13T15:45:08.525 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:45:08.525 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822658 2024-09-13T15:45:08.525 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:45:08.525 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:45:08.526 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822658 2024-09-13T15:45:08.526 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:45:08.526 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822658 2024-09-13T15:45:08.526 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822658' 2024-09-13T15:45:08.526 INFO:tasks.workunit.client.0.smithi112.stderr:466555: waiting osd.0 seq 154618822658 2024-09-13T15:45:08.526 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:45:08.526 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 154618822658 2024-09-13T15:45:08.526 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:45:08.526 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:45:08.526 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:45:08.526 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822659 -lt 154618822658 2024-09-13T15:45:08.724 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/stauntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:45:08.725 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822659 2024-09-13T15:45:08.725 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822659 2024-09-13T15:45:08.725 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822659' 2024-09-13T15:45:08.725 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:45:08.725 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:45:08.725 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672991 2024-09-13T15:45:08.725 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672991 2024-09-13T15:45:08.725 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822659 1-42949672991' 2024-09-13T15:45:08.725 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:45:08.725 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:45:08.725 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920772 2024-09-13T15:45:08.725 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920772 2024-09-13T15:45:08.726 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822659 1-42949672991 2-141733920772' 2024-09-13T15:45:08.726 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:45:08.726 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-13T15:45:08.726 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855364 2024-09-13T15:45:08.726 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855364 2024-09-13T15:45:08.726 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822659 1-42949672991 2-141733920772 3-150323855364' 2024-09-13T15:45:08.726 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:45:08.726 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822659 2024-09-13T15:45:08.726 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:45:08.726 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:45:08.726 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822659 2024-09-13T15:45:08.726 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:45:08.726 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822659 2024-09-13T15:45:08.727 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822659' 2024-09-13T15:45:08.727 INFO:tasks.workunit.client.0.smithi112.stderr:466561: waiting osd.0 seq 154618822659 2024-09-13T15:45:08.727 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:45:08.727 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 154618822659 2024-09-13T15:45:08.727 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:45:08.727 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:45:08.727 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:45:08.727 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822659 -lt 154618822659 2024-09-13T15:45:09.517 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/st //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:45:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:45:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:45:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:45:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:45:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:45:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T15:45:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:45:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:45:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:45:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T15:45:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T15:45:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T15:45:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T15:45:09.518 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T15:45:09.519 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T15:45:09.519 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T15:45:09.519 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T15:45:09.519 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T15:45:09.519 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T15:45:09.519 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T15:45:09.519 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T15:45:09.519 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T15:45:09.519 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T15:45:09.519 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T15:45:09.519 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:45:09.519 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:45:09.519 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:45:09.519 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:45:09.520 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:45:09.891 INFO:tasks.workunit.client.0.smithi112.stderr:4ndalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:45:09.891 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672990 2024-09-13T15:45:09.891 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:45:09.891 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:45:09.891 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672990 2024-09-13T15:45:09.892 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:45:09.892 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672990 2024-09-13T15:45:09.892 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-09-13T15:45:09.892 INFO:tasks.workunit.client.0.smithi112.stderr:466555: waiting osd.1 seq 42949672990 2024-09-13T15:45:09.892 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:45:09.892 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672991 -lt 42949672990 2024-09-13T15:45:09.892 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:45:09.892 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920771 2024-09-13T15:45:09.892 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:45:09.892 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:45:09.892 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920771 2024-09-13T15:45:09.892 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:45:09.892 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920771 2024-09-13T15:45:09.892 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2024-09-13T15:45:09.893 INFO:tasks.workunit.client.0.smithi112.stderr:466555: waiting osd.2 seq 141733920771 2024-09-13T15:45:09.893 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:45:09.893 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920771 2024-09-13T15:45:09.893 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:45:09.893 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:45:09.893 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-150323855363 2024-09-13T15:45:09.893 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T15:45:09.893 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-150323855363 2024-09-13T15:45:09.893 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:45:09.893 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855363 2024-09-13T15:45:09.893 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 150323855363' 2024-09-13T15:45:09.893 INFO:tasks.workunit.client.0.smithi112.stderr:466555: waiting osd.3 seq 150323855363 2024-09-13T15:45:09.894 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T15:45:09.894 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855364 -lt 150323855363 2024-09-13T15:45:09.894 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:45:09.894 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:45:09.894 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:45:09.894 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:45:10.223 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /homeandalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:45:10.223 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672991 2024-09-13T15:45:10.223 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:45:10.223 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:45:10.224 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672991 2024-09-13T15:45:10.224 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:45:10.224 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672991 2024-09-13T15:45:10.224 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2024-09-13T15:45:10.224 INFO:tasks.workunit.client.0.smithi112.stderr:466561: waiting osd.1 seq 42949672991 2024-09-13T15:45:10.224 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:45:10.224 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672991 -lt 42949672991 2024-09-13T15:45:10.224 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:45:10.224 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920772 2024-09-13T15:45:10.224 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:45:10.225 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:45:10.225 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920772 2024-09-13T15:45:10.225 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:45:10.225 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920772 2024-09-13T15:45:10.225 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920772' 2024-09-13T15:45:10.225 INFO:tasks.workunit.client.0.smithi112.stderr:466561: waiting osd.2 seq 141733920772 2024-09-13T15:45:10.225 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:45:10.225 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920772 -lt 141733920772 2024-09-13T15:45:10.225 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:45:10.225 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-150323855364 2024-09-13T15:45:10.225 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:45:10.225 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-13T15:45:10.226 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-150323855364 2024-09-13T15:45:10.226 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:45:10.226 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855364 2024-09-13T15:45:10.226 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 150323855364' 2024-09-13T15:45:10.226 INFO:tasks.workunit.client.0.smithi112.stderr:466561: waiting osd.3 seq 150323855364 2024-09-13T15:45:10.226 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-13T15:45:10.226 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855365 -lt 150323855364 2024-09-13T15:45:10.226 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:45:10.226 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:45:10.226 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:45:10.226 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:45:10.269 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /hom66559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:45:10.270 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:45:10.270 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:45:10.270 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:45:10.270 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:45:10.270 INFO:tasks.workunit.client.0.smithi112.stderr:466559: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:45:10.270 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:45:10.270 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:45:10.270 INFO:tasks.workunit.client.0.smithi112.stderr:466559: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:45:10.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T15:45:10.562 INFO:tasks.workunit.client.0.smithi112.stderr:/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:45:10.562 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:45:10.562 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:45:10.562 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:45:10.562 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:45:10.562 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:45:10.562 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:45:10.563 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:45:10.563 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:45:10.563 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:45:10.563 INFO:tasks.workunit.client.0.smithi112.stderr:466555: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:45:10.563 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:45:10.563 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:45:10.563 INFO:tasks.workunit.client.0.smithi112.stderr:466555: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:45:10.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T15:45:10.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T15:45:10.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 466553 2024-09-13T15:45:10.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-13T15:45:10.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 466556 2024-09-13T15:45:10.946 INFO:tasks.workunit.client.0.smithi112.stderr:e/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:45:10.947 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:45:10.947 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:45:10.947 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:45:10.947 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:45:10.947 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:45:10.947 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:45:10.947 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:45:10.947 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:45:10.947 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:45:10.947 INFO:tasks.workunit.client.0.smithi112.stderr:466561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:45:10.947 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:45:10.947 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:45:10.948 INFO:tasks.workunit.client.0.smithi112.stderr:466561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:45:10.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-13T15:45:10.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-13T15:45:10.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-13T15:45:10.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-13T15:45:10.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: return_code=0 2024-09-13T15:45:10.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:882: unfound_erasure_coded: '[' 0 -ne 0 ']' 2024-09-13T15:45:10.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: get_pg ecpool SOMETHING 2024-09-13T15:45:10.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-13T15:45:10.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T15:45:10.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-13T15:45:10.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T15:45:11.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: local pg=1.0 2024-09-13T15:45:11.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: unfound_erasure_coded: repair 1.0 2024-09-13T15:45:11.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-13T15:45:11.247 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-13T15:45:11.247 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:45:11.247 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:45:11.248 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:45:11.248 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:45:11.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T15:44:41.948147+0000 2024-09-13T15:45:11.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-13T15:45:11.916 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-09-13T15:45:11.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-13T15:44:41.948147+0000 2024-09-13T15:45:11.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T15:45:11.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:44:41.948147+0000 2024-09-13T15:45:11.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:45:11.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:45:11.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:45:11.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:45:11.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:45:11.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:45:11.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:45:11.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:45:12.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:44:41.948147+0000 '>' 2024-09-13T15:44:41.948147+0000 2024-09-13T15:45:12.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:45:13.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:45:13.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:45:13.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:45:13.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:45:13.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:45:13.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:45:13.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:45:13.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:44:41.948147+0000 '>' 2024-09-13T15:44:41.948147+0000 2024-09-13T15:45:13.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:45:14.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:45:14.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:45:14.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:45:14.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:45:14.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:45:14.572 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:45:14.572 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:45:14.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:44:41.948147+0000 '>' 2024-09-13T15:44:41.948147+0000 2024-09-13T15:45:14.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:45:15.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:45:15.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:45:15.900 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:45:15.900 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:45:15.900 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:45:15.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:45:15.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:45:16.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:44:41.948147+0000 '>' 2024-09-13T15:44:41.948147+0000 2024-09-13T15:45:16.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:45:17.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:45:17.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:45:17.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:45:17.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:45:17.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:45:17.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:45:17.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:45:17.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:44:41.948147+0000 '>' 2024-09-13T15:44:41.948147+0000 2024-09-13T15:45:17.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:45:18.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:45:18.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:45:18.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:45:18.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:45:18.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:45:18.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:45:18.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:45:18.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:44:41.948147+0000 '>' 2024-09-13T15:44:41.948147+0000 2024-09-13T15:45:18.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:45:19.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:45:19.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:45:19.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:45:19.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:45:19.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:45:19.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:45:19.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:45:20.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:45:12.855923+0000 '>' 2024-09-13T15:44:41.948147+0000 2024-09-13T15:45:20.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:45:20.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T15:45:22.232 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: seq 1 60 2024-09-13T15:45:22.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: for f in `seq 1 60` 2024-09-13T15:45:22.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: ceph -s 2024-09-13T15:45:22.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-13T15:45:22.615 INFO:tasks.workunit.client.0.smithi112.stdout: 1/1 objects unfound (100.000%) 2024-09-13T15:45:22.615 INFO:tasks.workunit.client.0.smithi112.stdout: 1/1 objects unfound (100.000%) 2024-09-13T15:45:22.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: break 2024-09-13T15:45:22.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: grep '4 up' 2024-09-13T15:45:22.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: ceph -s 2024-09-13T15:45:23.002 INFO:tasks.workunit.client.0.smithi112.stdout: osd: 4 osds: 4 up (since 18s), 4 in (since 90s) 2024-09-13T15:45:23.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: ceph -s 2024-09-13T15:45:23.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: grep '4 in' 2024-09-13T15:45:23.382 INFO:tasks.workunit.client.0.smithi112.stdout: osd: 4 osds: 4 up (since 18s), 4 in (since 90s) 2024-09-13T15:45:23.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: ceph -s 2024-09-13T15:45:23.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: grep '1/1 objects unfound' 2024-09-13T15:45:23.806 INFO:tasks.workunit.client.0.smithi112.stdout: 1/1 objects unfound (100.000%) 2024-09-13T15:45:23.806 INFO:tasks.workunit.client.0.smithi112.stdout: 1/1 objects unfound (100.000%) 2024-09-13T15:45:23.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-13T15:45:23.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:45:23.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:45:23.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:45:23.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:45:23.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:45:23.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:45:23.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:45:23.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:45:24.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:45:24.119 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:45:24.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:45:24.122 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:45:24.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:45:24.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:45:24.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:45:24.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:45:24.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:45:24.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:45:24.127 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:45:24.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:45:24.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:45:24.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:45:24.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:45:24.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:45:24.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:45:24.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:45:24.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:45:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:45:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-13T15:45:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-scrub-repair 0 2024-09-13T15:45:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-13T15:45:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-13T15:45:24.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-13T15:45:24.163 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:45:24.163 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:45:24.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:45:24.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:45:24.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:45:24.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:45:24.167 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:45:24.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:45:24.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:45:24.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:45:24.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:45:24.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:45:24.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:45:24.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:45:24.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:45:24.172 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:45:24.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:45:24.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-13T15:45:24.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-13T15:45:24.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:45:24.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:45:24.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.63282 2024-09-13T15:45:24.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.63282 2024-09-13T15:45:24.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:45:24.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:45:24.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-13T15:45:24.179 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T15:45:24.179 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T15:45:24.191 INFO:tasks.workunit:Running workunit scrub/osd-scrub-snaps.sh... 2024-09-13T15:45:24.192 DEBUG:teuthology.orchestra.run.smithi112:workunit test scrub/osd-scrub-snaps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d58f73069a8157a9ace12a0247ee3e8497707ea9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh 2024-09-13T15:45:24.251 INFO:tasks.workunit.client.0.smithi112.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-13T15:45:24.256 INFO:tasks.workunit.client.0.smithi112.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-13T15:45:24.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-13T15:45:24.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-13T15:45:24.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-13T15:45:24.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-13T15:45:24.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-13T15:45:24.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-13T15:45:24.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-13T15:45:24.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-13T15:45:24.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-scrub-snaps 2024-09-13T15:45:24.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:27: run: local dir=td/osd-scrub-snaps 2024-09-13T15:45:24.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:28: run: shift 2024-09-13T15:45:24.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: export CEPH_MON=127.0.0.1:7121 2024-09-13T15:45:24.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: CEPH_MON=127.0.0.1:7121 2024-09-13T15:45:24.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:31: run: export CEPH_ARGS 2024-09-13T15:45:24.258 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: uuidgen 2024-09-13T15:45:24.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: CEPH_ARGS+='--fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none ' 2024-09-13T15:45:24.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:33: run: CEPH_ARGS+='--mon-host=127.0.0.1:7121 ' 2024-09-13T15:45:24.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:35: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-13T15:45:24.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: set 2024-09-13T15:45:24.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-13T15:45:24.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: local 'funcs=TEST_scrub_snaps 2024-09-13T15:45:24.261 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_scrub_snaps_primary 2024-09-13T15:45:24.262 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_scrub_snaps_replica' 2024-09-13T15:45:24.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-09-13T15:45:24.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-09-13T15:45:24.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2024-09-13T15:45:24.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2024-09-13T15:45:24.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-13T15:45:24.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:45:24.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-13T15:45:24.263 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:45:24.263 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:45:24.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:45:24.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:45:24.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:45:24.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:45:24.265 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:45:24.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:45:24.265 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:45:24.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:45:24.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:45:24.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:45:24.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:45:24.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:45:24.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:45:24.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:45:24.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:45:24.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:45:24.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-13T15:45:24.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:45:24.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:45:24.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:45:24.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.471165 2024-09-13T15:45:24.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:45:24.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:45:24.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2024-09-13T15:45:24.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:45:24.274 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:45:24.274 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:45:24.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.471165 2024-09-13T15:45:24.277 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:45:24.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-13T15:45:24.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-13T15:45:24.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:45:24.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-09-13T15:45:24.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps td/osd-scrub-snaps 2024-09-13T15:45:24.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:171: TEST_scrub_snaps: local dir=td/osd-scrub-snaps 2024-09-13T15:45:24.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:172: TEST_scrub_snaps: local poolname=test 2024-09-13T15:45:24.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:173: TEST_scrub_snaps: local OBJS=16 2024-09-13T15:45:24.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:174: TEST_scrub_snaps: local OSDS=1 2024-09-13T15:45:24.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:176: TEST_scrub_snaps: TESTDATA=testdata.471165 2024-09-13T15:45:24.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:178: TEST_scrub_snaps: run_mon td/osd-scrub-snaps a --osd_pool_default_size=1 2024-09-13T15:45:24.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2024-09-13T15:45:24.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:45:24.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:45:24.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:45:24.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2024-09-13T15:45:24.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=1 2024-09-13T15:45:24.797 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:45:24.797 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:45:24.797 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:45:24.798 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:45:24.798 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:45:24.798 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:45:24.798 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:45:24.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 2024-09-13T15:45:25.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:45:25.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:45:25.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:45:25.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:45:25.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:45:25.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:45:25.067 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:45:25.067 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:45:25.067 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:45:25.067 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:45:25.068 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:45:25.068 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:45:25.068 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.471165/ceph-mon.a.asok 2024-09-13T15:45:25.068 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:45:25.068 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.471165/ceph-mon.a.asok config get fsid 2024-09-13T15:45:25.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:45:25.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:45:25.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:45:25.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:45:25.179 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:45:25.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:45:25.179 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:45:25.179 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:45:25.180 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:45:25.180 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:45:25.180 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:45:25.180 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.471165/ceph-mon.a.asok 2024-09-13T15:45:25.180 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:45:25.180 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.471165/ceph-mon.a.asok config get mon_host 2024-09-13T15:45:25.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:179: TEST_scrub_snaps: run_mgr td/osd-scrub-snaps x 2024-09-13T15:45:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2024-09-13T15:45:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:45:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:45:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:45:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2024-09-13T15:45:25.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:45:26.201 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:45:26.201 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:45:26.201 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:45:26.202 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:45:26.202 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:45:26.202 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:45:26.202 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:45:26.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:45:26.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:45:26.239 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: expr 1 - 1 2024-09-13T15:45:26.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: seq 0 0 2024-09-13T15:45:26.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:45:26.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:182: TEST_scrub_snaps: run_osd td/osd-scrub-snaps 0 2024-09-13T15:45:26.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2024-09-13T15:45:26.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:45:26.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:45:26.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:45:26.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-13T15:45:26.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-13T15:45:26.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:45:26.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:45:26.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:45:26.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-13T15:45:26.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-13T15:45:26.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:45:26.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:45:26.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-13T15:45:26.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:45:26.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:45:26.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:45:26.246 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:45:26.246 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:45:26.246 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:45:26.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:45:26.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:45:26.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:45:26.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:45:26.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:45:26.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-13T15:45:26.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-13T15:45:26.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:45:26.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:45:26.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:45:26.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:45:26.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:45:26.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:45:26.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-13T15:45:26.247 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:45:26.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b51effea-d051-4b42-b63d-ca1b039bf9c0 2024-09-13T15:45:26.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b51effea-d051-4b42-b63d-ca1b039bf9c0' 2024-09-13T15:45:26.247 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 b51effea-d051-4b42-b63d-ca1b039bf9c0 2024-09-13T15:45:26.248 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:45:26.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAWXuRmhSmaDxAAETZCe4P0iOVGQQuaq2cnRA== 2024-09-13T15:45:26.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAWXuRmhSmaDxAAETZCe4P0iOVGQQuaq2cnRA=="}' 2024-09-13T15:45:26.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b51effea-d051-4b42-b63d-ca1b039bf9c0 -i td/osd-scrub-snaps/0/new.json 2024-09-13T15:45:26.562 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:45:26.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-09-13T15:45:26.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAWXuRmhSmaDxAAETZCe4P0iOVGQQuaq2cnRA== --osd-uuid b51effea-d051-4b42-b63d-ca1b039bf9c0 2024-09-13T15:45:26.614 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:45:26.611+0000 7f59bd2ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:45:26.646 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:45:26.643+0000 7f59bd2ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:45:26.648 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:45:26.647+0000 7f59bd2ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:45:26.648 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:45:26.647+0000 7f59bd2ff7c0 -1 bdev(0x5572cc1d7c00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:45:26.649 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:45:26.647+0000 7f59bd2ff7c0 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-09-13T15:45:34.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-09-13T15:45:34.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:45:34.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:45:34.471 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:45:34.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:45:34.887 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:45:34.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:45:34.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:45:34.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:45:34.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:45:34.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:45:34.917 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:45:34.915+0000 7f24839647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:45:34.920 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:45:34.919+0000 7f24839647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:45:34.921 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:45:34.919+0000 7f24839647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:45:35.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:45:35.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:45:35.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:45:35.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:45:35.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:45:35.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:35.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:45:35.248 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:45:35.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:35.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:35.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:35.961 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:45:35.959+0000 7f24839647c0 -1 Falling back to public interface 2024-09-13T15:45:36.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:36.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:36.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:45:36.567 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:45:36.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:36.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:36.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:37.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:37.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:37.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:45:37.888 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:45:37.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:37.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:38.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:38.989 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:45:38.987+0000 7f24839647c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:45:39.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:39.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:39.220 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:45:39.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:45:39.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:39.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:39.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:40.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:40.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:40.693 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:45:40.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:45:40.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:40.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:41.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:42.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:42.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:42.121 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:45:42.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:45:42.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:42.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:42.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:43.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:43.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:43.445 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:45:43.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:45:43.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:43.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:43.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:44.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:44.859 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:45:44.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:44.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:45:44.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:44.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:45.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:46.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:46.290 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:45:46.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:46.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:45:46.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:46.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:46.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:47.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:47.611 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:45:47.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:47.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:45:47.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:47.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:48.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:49.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:49.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:49.133 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:45:49.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:45:49.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:49.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:49.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:50.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:50.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:50.471 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:45:50.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:45:50.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:50.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:50.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:51.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:51.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:51.871 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:45:51.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:45:51.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:51.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:52.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:53.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:53.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:53.230 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:45:53.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:45:53.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:53.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:53.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:54.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:54.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:54.768 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:45:54.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:45:54.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:54.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:55.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:56.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:56.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:56.279 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:45:56.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:45:56.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:56.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:56.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:57.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:57.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:57.603 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:45:57.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:45:57.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:57.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:57.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:45:58.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:45:58.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:45:58.959 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:45:58.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:45:58.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:45:58.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:45:59.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:46:00.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:46:00.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:46:00.944 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:46:00.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:46:00.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:46:00.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:46:01.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:46:02.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:46:02.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:46:02.467 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:46:02.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:46:02.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:46:02.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:46:03.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:46:04.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:46:04.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:46:04.004 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:46:04.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:46:04.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:46:04.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:46:04.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:46:05.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:46:05.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:46:05.795 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:46:05.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:46:05.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:46:05.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:46:06.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:46:07.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:46:07.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:46:07.262 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:46:07.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:46:07.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:46:07.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:46:07.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:46:08.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:46:08.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:46:08.980 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:46:08.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:46:08.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:46:08.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:46:09.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:46:10.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:46:10.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:46:10.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:46:10.474 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:46:10.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:46:10.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:46:10.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:46:11.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:46:11.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:46:11.886 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:46:11.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:46:11.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:46:11.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:46:12.239 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1541276904,v1:127.0.0.1:6803/1541276904] [v2:127.0.0.1:6804/1541276904,v1:127.0.0.1:6805/1541276904] exists,up b51effea-d051-4b42-b63d-ca1b039bf9c0 2024-09-13T15:46:12.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:46:12.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:46:12.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:46:12.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:186: TEST_scrub_snaps: ceph osd set noscrub 2024-09-13T15:46:12.676 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:46:12.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:187: TEST_scrub_snaps: ceph osd set nodeep-scrub 2024-09-13T15:46:13.371 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:46:13.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:190: TEST_scrub_snaps: create_pool test 1 1 2024-09-13T15:46:13.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-13T15:46:13.880 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'test' created 2024-09-13T15:46:13.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:46:14.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:191: TEST_scrub_snaps: wait_for_clean 2024-09-13T15:46:14.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:46:14.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:46:14.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:46:14.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:46:14.904 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:46:14.905 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:46:14.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:46:14.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:46:14.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:46:15.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:46:15.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:46:15.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:46:15.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:46:15.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:46:15.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:46:15.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T15:46:15.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:46:15.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:46:15.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:46:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836485 2024-09-13T15:46:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836485 2024-09-13T15:46:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836485' 2024-09-13T15:46:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:46:15.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836485 2024-09-13T15:46:15.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:46:15.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:46:15.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836485 2024-09-13T15:46:15.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:46:15.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836485 2024-09-13T15:46:15.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-13T15:46:15.542 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836485 2024-09-13T15:46:15.542 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:46:15.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836484 -lt 21474836485 2024-09-13T15:46:15.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:46:16.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:46:16.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:46:17.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836486 -lt 21474836485 2024-09-13T15:46:17.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:46:17.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:46:17.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:46:17.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:46:17.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:46:17.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:46:17.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:46:17.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:46:17.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:46:17.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:46:17.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:46:17.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:46:17.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:46:17.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:46:17.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:46:18.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:46:18.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:46:18.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:46:18.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: ceph osd dump 2024-09-13T15:46:18.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: awk '{ print $2 }' 2024-09-13T15:46:18.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: grep '^pool.*['\'']test['\'']' 2024-09-13T15:46:18.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: poolid=1 2024-09-13T15:46:18.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:194: TEST_scrub_snaps: dd if=/dev/urandom of=testdata.471165 bs=1032 count=1 2024-09-13T15:46:18.679 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T15:46:18.679 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T15:46:18.679 INFO:tasks.workunit.client.0.smithi112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000114551 s, 9.0 MB/s 2024-09-13T15:46:18.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: seq 1 16 2024-09-13T15:46:18.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:18.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj1 testdata.471165 2024-09-13T15:46:18.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:18.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj2 testdata.471165 2024-09-13T15:46:19.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:19.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj3 testdata.471165 2024-09-13T15:46:19.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:19.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj4 testdata.471165 2024-09-13T15:46:19.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:19.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj5 testdata.471165 2024-09-13T15:46:19.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:19.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj6 testdata.471165 2024-09-13T15:46:19.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:19.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj7 testdata.471165 2024-09-13T15:46:19.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:19.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj8 testdata.471165 2024-09-13T15:46:19.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:19.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj9 testdata.471165 2024-09-13T15:46:19.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:19.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj10 testdata.471165 2024-09-13T15:46:20.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:20.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj11 testdata.471165 2024-09-13T15:46:20.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:20.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj12 testdata.471165 2024-09-13T15:46:20.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:20.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj13 testdata.471165 2024-09-13T15:46:20.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:20.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj14 testdata.471165 2024-09-13T15:46:20.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:20.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj15 testdata.471165 2024-09-13T15:46:20.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-09-13T15:46:20.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj16 testdata.471165 2024-09-13T15:46:20.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: get_primary test obj1 2024-09-13T15:46:20.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-13T15:46:20.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T15:46:20.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-13T15:46:20.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:46:21.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: local primary=0 2024-09-13T15:46:21.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:202: TEST_scrub_snaps: create_scenario td/osd-scrub-snaps test testdata.471165 0 2024-09-13T15:46:21.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-09-13T15:46:21.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-09-13T15:46:21.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.471165 2024-09-13T15:46:21.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2024-09-13T15:46:21.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-09-13T15:46:21.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-09-13T15:46:21.473 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap1 2024-09-13T15:46:21.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=1 2024-09-13T15:46:21.478 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T15:46:21.478 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T15:46:21.478 INFO:tasks.workunit.client.0.smithi112.stderr:256 bytes copied, 0.000258463 s, 990 kB/s 2024-09-13T15:46:21.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.471165 2024-09-13T15:46:21.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.471165 2024-09-13T15:46:21.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.471165 2024-09-13T15:46:22.062 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-09-13T15:46:22.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:46:22.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.471165 2024-09-13T15:46:22.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:46:22.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.471165 2024-09-13T15:46:22.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:46:22.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.471165 2024-09-13T15:46:22.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:46:22.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.471165 2024-09-13T15:46:22.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:46:22.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.471165 2024-09-13T15:46:22.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:46:22.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.471165 2024-09-13T15:46:22.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:46:22.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.471165 2024-09-13T15:46:23.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:46:23.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.471165 2024-09-13T15:46:23.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:46:23.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.471165 2024-09-13T15:46:23.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-09-13T15:46:23.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-09-13T15:46:23.548 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap2 2024-09-13T15:46:23.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=2 2024-09-13T15:46:23.553 INFO:tasks.workunit.client.0.smithi112.stderr:2+0 records in 2024-09-13T15:46:23.553 INFO:tasks.workunit.client.0.smithi112.stderr:2+0 records out 2024-09-13T15:46:23.553 INFO:tasks.workunit.client.0.smithi112.stderr:512 bytes copied, 0.000208113 s, 2.5 MB/s 2024-09-13T15:46:23.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.471165 2024-09-13T15:46:23.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-09-13T15:46:23.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-09-13T15:46:23.983 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap3 2024-09-13T15:46:23.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=3 2024-09-13T15:46:23.989 INFO:tasks.workunit.client.0.smithi112.stderr:3+0 records in 2024-09-13T15:46:23.989 INFO:tasks.workunit.client.0.smithi112.stderr:3+0 records out 2024-09-13T15:46:23.989 INFO:tasks.workunit.client.0.smithi112.stderr:768 bytes copied, 0.000278383 s, 2.8 MB/s 2024-09-13T15:46:23.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.471165 2024-09-13T15:46:24.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-09-13T15:46:24.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-09-13T15:46:24.442 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap4 2024-09-13T15:46:24.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=4 2024-09-13T15:46:24.449 INFO:tasks.workunit.client.0.smithi112.stderr:4+0 records in 2024-09-13T15:46:24.449 INFO:tasks.workunit.client.0.smithi112.stderr:4+0 records out 2024-09-13T15:46:24.449 INFO:tasks.workunit.client.0.smithi112.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000167028 s, 6.1 MB/s 2024-09-13T15:46:24.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.471165 2024-09-13T15:46:24.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.471165 2024-09-13T15:46:24.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-09-13T15:46:24.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-09-13T15:46:25.175 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap5 2024-09-13T15:46:25.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-09-13T15:46:25.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-09-13T15:46:25.567 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap6 2024-09-13T15:46:25.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=6 2024-09-13T15:46:25.573 INFO:tasks.workunit.client.0.smithi112.stderr:6+0 records in 2024-09-13T15:46:25.573 INFO:tasks.workunit.client.0.smithi112.stderr:6+0 records out 2024-09-13T15:46:25.573 INFO:tasks.workunit.client.0.smithi112.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 0.00021485 s, 7.1 MB/s 2024-09-13T15:46:25.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.471165 2024-09-13T15:46:25.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-09-13T15:46:25.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-09-13T15:46:26.131 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap7 2024-09-13T15:46:26.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-09-13T15:46:26.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-09-13T15:46:26.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-09-13T15:46:26.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-09-13T15:46:26.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:46:26.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:46:26.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:46:26.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:46:26.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:46:26.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:46:26.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2024-09-13T15:46:27.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:46:27.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-09-13T15:46:30.086 INFO:tasks.workunit.client.0.smithi112.stdout:WARNING: only removing head with clones present 2024-09-13T15:46:30.086 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-13T15:46:30.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-13T15:46:30.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-09-13T15:46:31.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:46:31.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-13T15:46:32.723 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:c52c9666:::obj5:2# 2024-09-13T15:46:33.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-09-13T15:46:33.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-13T15:46:34.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:46:34.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:46:34.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-13T15:46:35.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-13T15:46:35.234 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:46:35.234 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-13T15:46:35.234 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:35.234 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:46:35.234 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-13T15:46:35.234 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-13T15:46:35.234 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-13T15:46:35.234 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-13T15:46:35.234 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-13T15:46:35.235 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-13T15:46:35.235 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:35.235 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-13T15:46:35.235 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-13T15:46:35.235 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-13T15:46:35.235 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-13T15:46:35.235 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:46:35.235 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:46:35.235 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:35.235 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:35.235 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:46:35.235 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:46:35.236 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:35.236 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:46:35.236 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:35.236 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:46:35.236 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:35.236 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:46:35.236 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:35.236 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:46:35.236 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:35.236 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:46:35.236 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:46:35.236 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:46:35.237 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:35.237 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:46:35.237 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:46:35.237 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:46:35.237 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:35.237 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:46:35.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-13T15:46:35.238 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:46:35.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-13T15:46:36.568 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:c52c9666:::obj5:1# 2024-09-13T15:46:37.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-13T15:46:38.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-13T15:46:38.214 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:46:38.215 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-13T15:46:38.215 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:38.215 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:46:38.215 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-13T15:46:38.215 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-13T15:46:38.215 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-13T15:46:38.215 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-13T15:46:38.215 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-13T15:46:38.215 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-13T15:46:38.215 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:38.215 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-13T15:46:38.215 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-13T15:46:38.216 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-13T15:46:38.216 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-13T15:46:38.216 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:46:38.216 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:46:38.216 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:38.216 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:38.216 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:46:38.216 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:46:38.216 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:38.216 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:46:38.216 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:38.216 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:46:38.216 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:38.217 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:46:38.217 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:38.217 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:46:38.217 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:38.217 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:46:38.217 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:46:38.217 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:46:38.217 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:38.217 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:46:38.217 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:46:38.217 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:46:38.217 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:38.217 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:46:38.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-13T15:46:38.218 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:46:38.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-13T15:46:38.219 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-13T15:46:38.219 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-09-13T15:46:39.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:46:39.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=18 2024-09-13T15:46:39.123 INFO:tasks.workunit.client.0.smithi112.stderr:18+0 records in 2024-09-13T15:46:39.123 INFO:tasks.workunit.client.0.smithi112.stderr:18+0 records out 2024-09-13T15:46:39.123 INFO:tasks.workunit.client.0.smithi112.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000285641 s, 16.1 MB/s 2024-09-13T15:46:39.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.471165 2024-09-13T15:46:40.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2024-09-13T15:46:41.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:46:41.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=15 2024-09-13T15:46:41.662 INFO:tasks.workunit.client.0.smithi112.stderr:15+0 records in 2024-09-13T15:46:41.662 INFO:tasks.workunit.client.0.smithi112.stderr:15+0 records out 2024-09-13T15:46:41.662 INFO:tasks.workunit.client.0.smithi112.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000216946 s, 17.7 MB/s 2024-09-13T15:46:41.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.471165 2024-09-13T15:46:43.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2024-09-13T15:46:43.515 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-09-13T15:46:44.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:46:44.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-13T15:46:45.786 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:0ee9ae15:::obj4:7# 2024-09-13T15:46:46.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-13T15:46:47.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-13T15:46:47.922 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-13T15:46:47.922 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:47.922 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:46:47.923 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-13T15:46:47.923 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-13T15:46:47.923 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-13T15:46:47.923 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-13T15:46:47.923 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-13T15:46:47.923 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-13T15:46:47.923 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:47.923 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-13T15:46:47.923 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-13T15:46:47.923 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-13T15:46:47.923 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-13T15:46:47.923 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:46:47.923 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:47.923 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:47.924 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:46:47.924 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:47.924 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:46:47.924 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:47.924 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:47.924 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:46:47.924 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:47.924 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:47.924 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:46:47.924 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:46:47.924 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:47.924 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:46:47.924 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:46:47.925 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:47.925 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:46:47.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-13T15:46:47.925 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:47.925 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:47.925 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:47.925 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:47.925 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:47.926 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:47.926 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:46:47.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-09-13T15:46:47.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2024-09-13T15:46:48.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:46:48.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-13T15:46:50.056 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:461f8b5e:::obj16:7# 2024-09-13T15:46:50.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-13T15:46:51.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-13T15:46:51.694 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-13T15:46:51.695 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:46:51.695 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-13T15:46:51.695 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-13T15:46:51.695 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-13T15:46:51.695 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-13T15:46:51.695 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-13T15:46:51.695 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-13T15:46:51.696 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:51.696 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-13T15:46:51.696 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-13T15:46:51.696 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-13T15:46:51.696 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-13T15:46:51.697 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:46:51.697 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:51.697 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:46:51.697 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:46:51.697 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:51.697 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:46:51.697 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:46:51.698 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:46:51.698 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:46:51.698 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:46:51.698 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:46:51.698 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:46:51.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-13T15:46:51.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-13T15:46:51.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2024-09-13T15:46:52.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:46:52.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-09-13T15:46:54.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:46:54.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-09-13T15:46:54.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:46:54.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=7 2024-09-13T15:46:54.450 INFO:tasks.workunit.client.0.smithi112.stderr:7+0 records in 2024-09-13T15:46:54.450 INFO:tasks.workunit.client.0.smithi112.stderr:7+0 records out 2024-09-13T15:46:54.450 INFO:tasks.workunit.client.0.smithi112.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 0.000321182 s, 5.6 MB/s 2024-09-13T15:46:54.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.471165 2024-09-13T15:46:56.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2024-09-13T15:46:56.728 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:46:57.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:46:57.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-09-13T15:46:59.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2024-09-13T15:46:59.488 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:46:59.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:46:59.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-09-13T15:47:02.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2024-09-13T15:47:02.411 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:47:02.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:47:02.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-09-13T15:47:04.555 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2024-09-13T15:47:04.903 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:47:05.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:47:05.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-09-13T15:47:07.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2024-09-13T15:47:07.489 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:47:08.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:47:08.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-09-13T15:47:10.366 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2024-09-13T15:47:10.720 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:47:11.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:47:11.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-09-13T15:47:12.969 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2024-09-13T15:47:13.371 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:47:13.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:47:13.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-09-13T15:47:15.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2024-09-13T15:47:15.819 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:47:16.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:47:16.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-09-13T15:47:18.186 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2024-09-13T15:47:18.770 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:47:19.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:47:19.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-09-13T15:47:21.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-09-13T15:47:21.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2024-09-13T15:47:21.855 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:47:22.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:47:22.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-09-13T15:47:24.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-09-13T15:47:24.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-09-13T15:47:24.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:204: TEST_scrub_snaps: rm -f testdata.471165 2024-09-13T15:47:24.296 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: expr 1 - 1 2024-09-13T15:47:24.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: seq 0 0 2024-09-13T15:47:24.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:47:24.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:208: TEST_scrub_snaps: activate_osd td/osd-scrub-snaps 0 2024-09-13T15:47:24.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2024-09-13T15:47:24.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:47:24.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:47:24.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:47:24.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-13T15:47:24.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-13T15:47:24.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:47:24.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:47:24.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:47:24.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-13T15:47:24.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-13T15:47:24.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:47:24.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:47:24.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-13T15:47:24.302 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:47:24.302 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:47:24.303 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:47:24.303 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:47:24.303 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:47:24.303 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:47:24.303 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:47:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:47:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:47:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-13T15:47:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-13T15:47:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:47:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:47:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:47:24.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:47:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:47:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:47:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-13T15:47:24.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:47:24.305 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:47:24.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:47:24.306 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-09-13T15:47:24.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:47:24.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:47:24.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:47:24.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:47:24.334 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:47:24.332+0000 7f98f6fae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:47:24.337 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:47:24.336+0000 7f98f6fae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:47:24.339 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:47:24.340+0000 7f98f6fae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:47:24.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:47:24.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:47:24.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:47:24.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:47:24.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:47:24.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:47:24.633 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:47:24.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:47:24.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:47:24.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:47:24.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:47:25.616 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:47:25.612+0000 7f98f6fae7c0 -1 Falling back to public interface 2024-09-13T15:47:25.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:47:25.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:47:25.963 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:47:25.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:47:25.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:47:25.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:47:26.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:47:27.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:47:27.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:47:27.278 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:47:27.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:47:27.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:47:27.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:47:27.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:47:28.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:47:28.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:47:28.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:47:28.607 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:47:28.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:47:28.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:47:28.612 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:47:28.612+0000 7f98f6fae7c0 -1 osd.0 18 log_to_monitors true 2024-09-13T15:47:28.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:47:29.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:47:29.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:47:29.948 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:47:29.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:47:29.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:47:29.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:47:30.266 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6802/1424654433,v1:127.0.0.1:6803/1424654433] [v2:127.0.0.1:6804/1424654433,v1:127.0.0.1:6805/1424654433] exists,up b51effea-d051-4b42-b63d-ca1b039bf9c0 2024-09-13T15:47:30.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:47:30.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:47:30.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:47:30.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:210: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 25 2024-09-13T15:47:30.400 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:47:30.400 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-13T15:47:30.400 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:47:30.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:211: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 5 2024-09-13T15:47:30.557 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:47:30.557 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-13T15:47:30.557 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:47:30.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:212: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2024-09-13T15:47:30.691 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:47:30.691 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-09-13T15:47:30.691 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:47:30.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:213: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2024-09-13T15:47:30.832 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:47:30.833 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-09-13T15:47:30.833 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:47:30.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:216: TEST_scrub_snaps: wait_for_clean 2024-09-13T15:47:30.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:47:30.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:47:30.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:47:30.847 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:47:30.848 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:47:30.848 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:47:30.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:47:30.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:47:30.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:47:30.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:47:30.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:47:30.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:47:30.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:47:30.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:47:30.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:47:31.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-13T15:47:31.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:47:31.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:47:31.241 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:47:31.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=90194313218 2024-09-13T15:47:31.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 90194313218 2024-09-13T15:47:31.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-90194313218' 2024-09-13T15:47:31.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:47:31.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-90194313218 2024-09-13T15:47:31.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:47:31.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:47:31.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-90194313218 2024-09-13T15:47:31.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:47:31.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=90194313218 2024-09-13T15:47:31.378 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 90194313218 2024-09-13T15:47:31.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 90194313218' 2024-09-13T15:47:31.379 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:47:31.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 90194313218 2024-09-13T15:47:31.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:47:32.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:47:32.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:47:32.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 90194313218 2024-09-13T15:47:32.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:47:33.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T15:47:33.996 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:47:34.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 90194313218 -lt 90194313218 2024-09-13T15:47:34.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:47:34.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:47:34.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:47:34.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:47:34.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:47:34.692 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:47:34.692 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:47:34.692 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:47:34.692 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:47:34.692 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:47:34.692 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:47:34.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:47:34.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:47:34.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:47:34.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:47:35.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:47:35.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:47:35.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:47:35.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:218: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_max 2024-09-13T15:47:35.504 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:47:35.505 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_shallow_scrub_chunk_max": "25" 2024-09-13T15:47:35.505 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:47:35.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:219: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_min 2024-09-13T15:47:35.653 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:47:35.653 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_shallow_scrub_chunk_min": "5" 2024-09-13T15:47:35.654 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:47:35.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:220: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max_seconds 2024-09-13T15:47:35.784 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:47:35.784 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_pg_stat_report_interval_max_seconds": "1" 2024-09-13T15:47:35.784 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:47:35.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:221: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max_epochs 2024-09-13T15:47:35.937 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:47:35.937 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_pg_stat_report_interval_max_epochs": "1" 2024-09-13T15:47:35.937 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:47:35.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:222: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_max 2024-09-13T15:47:36.068 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:47:36.068 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_chunk_max": "25" 2024-09-13T15:47:36.068 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:47:36.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:223: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_min 2024-09-13T15:47:36.213 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:47:36.213 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_chunk_min": "5" 2024-09-13T15:47:36.213 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:47:36.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:225: TEST_scrub_snaps: local pgid=1.0 2024-09-13T15:47:36.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:226: TEST_scrub_snaps: pg_scrub 1.0 2024-09-13T15:47:36.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-13T15:47:36.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-13T15:47:36.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:47:36.228 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:47:36.228 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:47:36.228 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:47:36.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-13T15:46:13.801734+0000 2024-09-13T15:47:36.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-13T15:47:36.846 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-13T15:47:36.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-13T15:46:13.801734+0000 2024-09-13T15:47:36.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T15:47:36.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:46:13.801734+0000 2024-09-13T15:47:36.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:47:36.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:47:36.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:47:36.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:47:36.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:47:36.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:47:36.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:47:36.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:47:37.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:46:13.801734+0000 '>' 2024-09-13T15:46:13.801734+0000 2024-09-13T15:47:37.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:47:38.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:47:38.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:47:38.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:47:38.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:47:38.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:47:38.159 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:47:38.159 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:47:38.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:46:13.801734+0000 '>' 2024-09-13T15:46:13.801734+0000 2024-09-13T15:47:38.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:47:39.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:47:39.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:47:39.469 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:47:39.469 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:47:39.469 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:47:39.470 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:47:39.470 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:47:39.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:47:37.658799+0000 '>' 2024-09-13T15:46:13.801734+0000 2024-09-13T15:47:39.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:47:39.783 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:39.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: wc -l 2024-09-13T15:47:39.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: test 2 = 2 2024-09-13T15:47:39.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:232: TEST_scrub_snaps: rados list-inconsistent-pg test 2024-09-13T15:47:39.811 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: jq '. | length' td/osd-scrub-snaps/json 2024-09-13T15:47:39.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: test 1 = 1 2024-09-13T15:47:39.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: jq -r '.[0]' td/osd-scrub-snaps/json 2024-09-13T15:47:39.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: test 1.0 = 1.0 2024-09-13T15:47:39.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:238: TEST_scrub_snaps: rados list-inconsistent-obj 1.0 2024-09-13T15:47:39.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: jq .inconsistents 2024-09-13T15:47:39.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-13T15:47:39.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:250: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2024-09-13T15:47:39.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:250: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-13T15:47:39.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:251: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-13T15:47:39.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-13T15:47:39.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:253: TEST_scrub_snaps: rados list-inconsistent-snapset 1.0 2024-09-13T15:47:39.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:255: TEST_scrub_snaps: jq .inconsistents 2024-09-13T15:47:39.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:255: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-13T15:47:40.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2024-09-13T15:47:40.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-13T15:47:40.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:635: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-13T15:47:40.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-13T15:47:40.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:636: TEST_scrub_snaps: test no = yes 2024-09-13T15:47:40.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:641: TEST_scrub_snaps: test '' = yes 2024-09-13T15:47:40.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: find td/osd-scrub-snaps 2024-09-13T15:47:40.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: grep 'osd[^/]*\.pid' 2024-09-13T15:47:40.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: pidfiles=td/osd-scrub-snaps/osd.0.pid 2024-09-13T15:47:40.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:647: TEST_scrub_snaps: pids= 2024-09-13T15:47:40.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:648: TEST_scrub_snaps: for pidfile in ${pidfiles} 2024-09-13T15:47:40.037 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:650: TEST_scrub_snaps: cat td/osd-scrub-snaps/osd.0.pid 2024-09-13T15:47:40.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:650: TEST_scrub_snaps: pids+='486994 ' 2024-09-13T15:47:40.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: ERRORS=0 2024-09-13T15:47:40.038 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: seq 1 7 2024-09-13T15:47:40.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-13T15:47:40.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap1 2024-09-13T15:47:40.252 INFO:tasks.workunit.client.0.smithi112.stdout:removed pool test snap snap1 2024-09-13T15:47:40.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-13T15:47:40.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap2 2024-09-13T15:47:40.516 INFO:tasks.workunit.client.0.smithi112.stdout:removed pool test snap snap2 2024-09-13T15:47:40.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-13T15:47:40.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap3 2024-09-13T15:47:40.834 INFO:tasks.workunit.client.0.smithi112.stdout:removed pool test snap snap3 2024-09-13T15:47:40.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-13T15:47:40.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap4 2024-09-13T15:47:41.118 INFO:tasks.workunit.client.0.smithi112.stdout:removed pool test snap snap4 2024-09-13T15:47:41.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-13T15:47:41.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap5 2024-09-13T15:47:41.439 INFO:tasks.workunit.client.0.smithi112.stdout:removed pool test snap snap5 2024-09-13T15:47:41.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-13T15:47:41.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap6 2024-09-13T15:47:41.712 INFO:tasks.workunit.client.0.smithi112.stdout:removed pool test snap snap6 2024-09-13T15:47:41.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2024-09-13T15:47:41.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap7 2024-09-13T15:47:41.996 INFO:tasks.workunit.client.0.smithi112.stdout:removed pool test snap snap7 2024-09-13T15:47:42.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:659: TEST_scrub_snaps: sleep 5 2024-09-13T15:47:47.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:660: TEST_scrub_snaps: local -i loop=0 2024-09-13T15:47:47.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: ceph pg dump pgs 2024-09-13T15:47:47.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: grep -q snaptrim 2024-09-13T15:47:47.309 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:47:47.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: ceph pg dump pgs 2024-09-13T15:47:47.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: grep -q snaptrim_error 2024-09-13T15:47:47.628 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:47:47.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:665: TEST_scrub_snaps: break 2024-09-13T15:47:47.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:674: TEST_scrub_snaps: ceph pg dump pgs 2024-09-13T15:47:47.934 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T15:47:47.934 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 35 0 0 0 0 23416 0 0 60 0 60 active+clean+inconsistent+snaptrim_error 2024-09-13T15:47:42.040303+0000 23'60 29:139 [0] 0 [0] 0 18'56 2024-09-13T15:47:37.658799+0000 0'0 2024-09-13T15:46:13.801734+0000 7 1 periodic scrub scheduled @ 2024-09-14T16:52:00.926359+0000 33 0 2024-09-13T15:47:47.935 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T15:47:47.935 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T15:47:47.935 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:47:47.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:676: TEST_scrub_snaps: for pid in $pids 2024-09-13T15:47:47.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:678: TEST_scrub_snaps: kill -0 486994 2024-09-13T15:47:47.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:685: TEST_scrub_snaps: kill_daemons td/osd-scrub-snaps 2024-09-13T15:47:47.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:47:47.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:47:47.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:47:47.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:47:47.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:47:53.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:47:53.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:687: TEST_scrub_snaps: declare -a err_strings 2024-09-13T15:47:53.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:688: TEST_scrub_snaps: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' 2024-09-13T15:47:53.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:689: TEST_scrub_snaps: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' 2024-09-13T15:47:53.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:690: TEST_scrub_snaps: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' 2024-09-13T15:47:53.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:691: TEST_scrub_snaps: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' 2024-09-13T15:47:53.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:692: TEST_scrub_snaps: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' 2024-09-13T15:47:53.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:693: TEST_scrub_snaps: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' 2024-09-13T15:47:53.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:694: TEST_scrub_snaps: err_strings[6]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' 2024-09-13T15:47:53.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:695: TEST_scrub_snaps: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' 2024-09-13T15:47:53.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:696: TEST_scrub_snaps: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' 2024-09-13T15:47:53.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:697: TEST_scrub_snaps: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' 2024-09-13T15:47:53.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:698: TEST_scrub_snaps: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' 2024-09-13T15:47:53.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:699: TEST_scrub_snaps: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' 2024-09-13T15:47:53.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' 2024-09-13T15:47:53.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:701: TEST_scrub_snaps: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' 2024-09-13T15:47:53.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' 2024-09-13T15:47:53.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:703: TEST_scrub_snaps: err_strings[15]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' 2024-09-13T15:47:53.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:704: TEST_scrub_snaps: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' 2024-09-13T15:47:53.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:705: TEST_scrub_snaps: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' 2024-09-13T15:47:53.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:706: TEST_scrub_snaps: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' 2024-09-13T15:47:53.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:707: TEST_scrub_snaps: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' 2024-09-13T15:47:53.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:708: TEST_scrub_snaps: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2024-09-13T15:47:53.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:709: TEST_scrub_snaps: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' 2024-09-13T15:47:53.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-09-13T15:47:53.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-09-13T15:47:53.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:47:53.338 INFO:tasks.workunit.client.0.smithi112.stdout:TEST PASSED 2024-09-13T15:47:53.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:721: TEST_scrub_snaps: '[' 0 '!=' 0 ']' 2024-09-13T15:47:53.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:727: TEST_scrub_snaps: echo 'TEST PASSED' 2024-09-13T15:47:53.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:728: TEST_scrub_snaps: return 0 2024-09-13T15:47:53.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-09-13T15:47:53.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-13T15:47:53.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:47:53.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-13T15:47:53.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:47:53.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:47:53.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:47:53.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:47:53.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:47:53.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:47:53.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:47:53.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:47:53.349 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:47:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:47:53.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:47:53.350 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:47:53.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:47:53.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:47:53.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:47:53.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:47:53.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:47:53.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:47:53.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-13T15:47:53.368 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:47:53.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:47:53.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:47:53.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.471165 2024-09-13T15:47:53.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:47:53.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:47:53.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-09-13T15:47:53.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-09-13T15:47:53.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2024-09-13T15:47:53.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2024-09-13T15:47:53.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-13T15:47:53.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:47:53.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-13T15:47:53.372 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:47:53.372 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:47:53.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:47:53.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:47:53.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:47:53.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:47:53.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:47:53.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:47:53.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:47:53.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:47:53.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:47:53.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:47:53.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:47:53.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:47:53.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:47:53.379 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:47:53.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:47:53.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:47:53.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-13T15:47:53.383 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:47:53.383 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:47:53.383 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:47:53.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.471165 2024-09-13T15:47:53.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:47:53.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:47:53.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2024-09-13T15:47:53.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:47:53.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:47:53.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:47:53.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.471165 2024-09-13T15:47:53.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:47:53.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T15:47:53.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:47:53.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-09-13T15:47:53.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_primary td/osd-scrub-snaps 2024-09-13T15:47:53.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1177: TEST_scrub_snaps_primary: local dir=td/osd-scrub-snaps 2024-09-13T15:47:53.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1178: TEST_scrub_snaps_primary: ORIG_ARGS='--fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-13T15:47:53.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1179: TEST_scrub_snaps_primary: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T15:47:53.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1180: TEST_scrub_snaps_primary: _scrub_snaps_multi td/osd-scrub-snaps primary 2024-09-13T15:47:53.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2024-09-13T15:47:53.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local poolname=test 2024-09-13T15:47:53.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local OBJS=16 2024-09-13T15:47:53.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:735: _scrub_snaps_multi: local OSDS=2 2024-09-13T15:47:53.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: local which=primary 2024-09-13T15:47:53.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: TESTDATA=testdata.471165 2024-09-13T15:47:53.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2024-09-13T15:47:53.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2024-09-13T15:47:53.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:47:53.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:47:53.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:47:53.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2024-09-13T15:47:53.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2024-09-13T15:47:53.942 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:47:53.943 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:47:53.943 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:47:53.943 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:47:53.943 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:47:53.943 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:47:53.944 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:47:53.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-13T15:47:54.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:47:54.203 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:47:54.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:47:54.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:47:54.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:47:54.205 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:47:54.205 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:47:54.205 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:47:54.205 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:47:54.205 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:47:54.205 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:47:54.205 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:47:54.206 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.471165/ceph-mon.a.asok 2024-09-13T15:47:54.206 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:47:54.206 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.471165/ceph-mon.a.asok config get fsid 2024-09-13T15:47:54.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:47:54.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:47:54.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:47:54.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:47:54.323 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:47:54.323 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:47:54.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:47:54.324 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:47:54.324 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:47:54.324 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:47:54.324 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:47:54.324 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.471165/ceph-mon.a.asok 2024-09-13T15:47:54.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:47:54.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.471165/ceph-mon.a.asok config get mon_host 2024-09-13T15:47:54.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2024-09-13T15:47:54.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2024-09-13T15:47:54.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:47:54.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:47:54.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:47:54.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2024-09-13T15:47:54.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:47:55.345 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:47:55.345 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:47:55.345 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:47:55.346 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:47:55.346 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:47:55.346 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:47:55.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:47:55.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:47:55.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:47:55.408 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: expr 2 - 1 2024-09-13T15:47:55.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: seq 0 1 2024-09-13T15:47:55.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:47:55.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2024-09-13T15:47:55.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2024-09-13T15:47:55.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:47:55.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:47:55.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:47:55.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-13T15:47:55.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T15:47:55.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:47:55.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:47:55.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:47:55.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-13T15:47:55.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-13T15:47:55.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:47:55.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:47:55.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-13T15:47:55.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:47:55.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:47:55.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:47:55.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:47:55.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-13T15:47:55.415 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:47:55.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ff85374b-00bd-419c-a965-df40eb1e79c7 2024-09-13T15:47:55.415 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 ff85374b-00bd-419c-a965-df40eb1e79c7 2024-09-13T15:47:55.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ff85374b-00bd-419c-a965-df40eb1e79c7' 2024-09-13T15:47:55.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:47:55.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCrXuRm4zJaGhAA0qnKbYuI0rIZgi96KWcZTQ== 2024-09-13T15:47:55.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCrXuRm4zJaGhAA0qnKbYuI0rIZgi96KWcZTQ=="}' 2024-09-13T15:47:55.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ff85374b-00bd-419c-a965-df40eb1e79c7 -i td/osd-scrub-snaps/0/new.json 2024-09-13T15:47:55.742 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:47:55.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-09-13T15:47:55.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCrXuRm4zJaGhAA0qnKbYuI0rIZgi96KWcZTQ== --osd-uuid ff85374b-00bd-419c-a965-df40eb1e79c7 2024-09-13T15:47:55.808 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:47:55.807+0000 7fe9180327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:47:55.824 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:47:55.823+0000 7fe9180327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:47:55.825 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:47:55.823+0000 7fe9180327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:47:55.826 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:47:55.823+0000 7fe9180327c0 -1 bdev(0x5625b6745c00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:47:55.826 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:47:55.823+0000 7fe9180327c0 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-09-13T15:48:03.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-09-13T15:48:03.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:48:03.842 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:48:03.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:48:03.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:48:04.259 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:48:04.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:48:04.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:48:04.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:48:04.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:48:04.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:48:04.305 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:48:04.303+0000 7fc53194b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:48:04.338 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:48:04.335+0000 7fc53194b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:48:04.339 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:48:04.339+0000 7fc53194b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:48:04.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:48:04.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:48:04.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:48:04.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:48:04.633 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:48:04.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:48:04.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:04.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:48:04.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:04.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:04.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:05.645 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:48:05.643+0000 7fc53194b7c0 -1 Falling back to public interface 2024-09-13T15:48:05.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:05.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:05.992 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:48:05.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:48:05.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:05.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:06.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:07.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:07.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:07.343 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:48:07.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:48:07.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:07.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:07.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:08.664 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:48:08.663+0000 7fc53194b7c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:48:08.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:08.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:08.716 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:48:08.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:48:08.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:08.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:09.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:10.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:10.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:10.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:48:10.321 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:48:10.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:10.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:10.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:11.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:11.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:11.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:48:11.965 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:48:11.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:11.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:12.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:13.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:13.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:13.101 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:48:13.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:48:13.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:13.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:13.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:14.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:14.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:14.457 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:48:14.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:48:14.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:14.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:14.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:48:15.992 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:48:15.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:15.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:16.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:17.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:17.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:17.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:48:17.338 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:48:17.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:17.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:17.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:18.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:18.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:18.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:48:18.789 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:48:18.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:18.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:19.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:20.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:20.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:20.143 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:48:20.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:48:20.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:20.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:20.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:21.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:21.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:21.498 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:48:21.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:48:21.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:21.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:21.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:22.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:22.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:22.964 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:48:22.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:48:22.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:22.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:23.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:24.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:24.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:24.322 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:48:24.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:48:24.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:24.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:24.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:25.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:25.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:25.812 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:48:25.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:48:25.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:25.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:26.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:27.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:27.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:27.162 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:48:27.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:48:27.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:27.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:27.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:28.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:28.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:28.636 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:48:28.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:48:28.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:28.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:29.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:30.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:30.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:30.689 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:48:30.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:48:30.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:30.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:31.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:32.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:32.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:32.438 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:48:32.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:48:32.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:32.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:33.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:34.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:34.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:34.510 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:48:34.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:48:34.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:34.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:35.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:36.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:36.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:36.272 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:48:36.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:48:36.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:36.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:36.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:37.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:37.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:37.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:48:37.987 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:48:37.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:37.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:38.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:39.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:39.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:39.728 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:48:39.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:48:39.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:39.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:40.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:41.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:41.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:41.219 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:48:41.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:48:41.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:41.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:48:41.572 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2541390033,v1:127.0.0.1:6803/2541390033] [v2:127.0.0.1:6804/2541390033,v1:127.0.0.1:6805/2541390033] exists,up ff85374b-00bd-419c-a965-df40eb1e79c7 2024-09-13T15:48:41.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:48:41.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:48:41.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:48:41.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:48:41.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2024-09-13T15:48:41.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2024-09-13T15:48:41.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:48:41.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T15:48:41.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:48:41.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-13T15:48:41.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T15:48:41.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:48:41.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:48:41.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:48:41.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-13T15:48:41.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-13T15:48:41.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:48:41.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:48:41.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-13T15:48:41.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:48:41.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:48:41.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:48:41.576 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:48:41.576 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:48:41.576 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:48:41.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:48:41.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:48:41.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:48:41.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:48:41.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:48:41.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-13T15:48:41.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-13T15:48:41.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:48:41.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:48:41.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:48:41.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:48:41.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:48:41.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:48:41.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-13T15:48:41.578 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:48:41.578 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 c3dd738e-e589-48a3-b2dd-5916f6bdb2a9 2024-09-13T15:48:41.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c3dd738e-e589-48a3-b2dd-5916f6bdb2a9 2024-09-13T15:48:41.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 c3dd738e-e589-48a3-b2dd-5916f6bdb2a9' 2024-09-13T15:48:41.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:48:41.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDZXuRm1jBMJBAAYIXdp6lzI2y2sv+oLNrKOQ== 2024-09-13T15:48:41.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDZXuRm1jBMJBAAYIXdp6lzI2y2sv+oLNrKOQ=="}' 2024-09-13T15:48:41.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c3dd738e-e589-48a3-b2dd-5916f6bdb2a9 -i td/osd-scrub-snaps/1/new.json 2024-09-13T15:48:42.169 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:48:42.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/1/new.json 2024-09-13T15:48:42.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZXuRm1jBMJBAAYIXdp6lzI2y2sv+oLNrKOQ== --osd-uuid c3dd738e-e589-48a3-b2dd-5916f6bdb2a9 2024-09-13T15:48:42.242 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:48:42.238+0000 7fc97e6b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:48:42.296 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:48:42.294+0000 7fc97e6b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:48:42.297 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:48:42.294+0000 7fc97e6b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:48:42.298 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:48:42.294+0000 7fc97e6b27c0 -1 bdev(0x55835718e800 td/osd-scrub-snaps/1/block) open stat got: (1) Operation not permitted 2024-09-13T15:48:42.298 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:48:42.294+0000 7fc97e6b27c0 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2024-09-13T15:48:50.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2024-09-13T15:48:50.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:48:50.893 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T15:48:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T15:48:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:48:51.320 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:48:51.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T15:48:51.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:48:51.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:48:51.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:48:51.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:48:51.360 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:48:51.358+0000 7f7a91c747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:48:51.394 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:48:51.390+0000 7f7a91c747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:48:51.395 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:48:51.394+0000 7f7a91c747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:48:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T15:48:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:48:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:48:51.714 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:48:51.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:48:51.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:48:51.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:51.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:48:51.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:51.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:48:52.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:52.405 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:48:52.402+0000 7f7a91c747c0 -1 Falling back to public interface 2024-09-13T15:48:53.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:53.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:53.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:48:53.091 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:48:53.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:53.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:48:53.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:54.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:54.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:54.445 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:48:54.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:48:54.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:54.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:48:54.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:55.480 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:48:55.477+0000 7f7a91c747c0 -1 osd.1 0 log_to_monitors true 2024-09-13T15:48:55.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:55.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:55.813 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:48:55.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:48:55.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:48:55.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:56.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:57.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:58.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:58.185 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:48:58.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:48:58.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:58.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:48:58.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:48:58.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:48:58.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:48:58.908 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:48:58.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:48:58.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:48:58.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:48:59.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:00.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:00.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:00.432 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:49:00.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:49:00.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:00.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:00.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:01.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:01.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:01.969 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:49:01.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:49:01.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:01.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:02.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:03.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:03.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:03.317 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:49:03.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:49:03.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:03.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:03.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:04.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:04.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:04.687 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:49:04.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:49:04.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:04.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:05.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:06.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:06.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:06.039 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:49:06.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:49:06.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:06.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:06.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:07.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:07.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:07.514 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:49:07.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:49:07.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:07.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:08.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:09.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:09.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:09.041 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:49:09.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:49:09.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:09.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:09.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:10.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:10.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:10.394 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:49:10.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:49:10.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:10.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:10.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:11.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:11.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:11.764 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:49:11.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:49:11.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:11.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:12.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:13.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:13.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:13.116 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:49:13.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:49:13.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:13.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:13.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:14.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:14.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:14.540 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:49:14.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:49:14.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:14.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:15.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:16.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:16.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:16.274 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:49:16.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:49:16.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:16.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:16.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:17.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:17.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:17.727 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:49:17.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:49:17.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:17.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:18.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:19.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:19.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:19.270 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:49:19.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:49:19.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:19.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:19.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:20.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:20.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:20.645 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:49:20.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:49:20.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:20.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:21.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:22.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:22.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:22.703 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:49:22.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:49:22.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:22.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:23.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:24.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:24.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:24.797 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:49:24.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:49:24.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:24.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:25.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:26.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:26.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:26.880 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:49:26.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:49:26.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:26.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:27.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:49:28.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:49:28.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:49:28.865 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:49:28.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:49:28.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:49:28.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:49:29.216 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1736231383,v1:127.0.0.1:6811/1736231383] [v2:127.0.0.1:6812/1736231383,v1:127.0.0.1:6813/1736231383] exists,up c3dd738e-e589-48a3-b2dd-5916f6bdb2a9 2024-09-13T15:49:29.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:49:29.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:49:29.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:49:29.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:748: _scrub_snaps_multi: ceph osd set noscrub 2024-09-13T15:49:29.724 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:49:29.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:749: _scrub_snaps_multi: ceph osd set nodeep-scrub 2024-09-13T15:49:30.234 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:49:30.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: create_pool test 1 1 2024-09-13T15:49:30.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-13T15:49:30.845 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'test' created 2024-09-13T15:49:30.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:49:31.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: wait_for_clean 2024-09-13T15:49:31.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:49:31.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:49:31.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:49:31.869 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:49:31.869 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:49:31.870 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:49:31.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:49:31.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:49:31.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:49:31.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:49:31.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:49:31.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:49:31.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:49:31.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:49:31.993 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:49:32.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:49:32.394 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:49:32.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:49:32.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:49:32.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:49:32.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836496 2024-09-13T15:49:32.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836496 2024-09-13T15:49:32.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836496' 2024-09-13T15:49:32.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:49:32.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:49:32.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672972 2024-09-13T15:49:32.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672972 2024-09-13T15:49:32.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836496 1-42949672972' 2024-09-13T15:49:32.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:49:32.773 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836496 2024-09-13T15:49:32.773 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:49:32.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:49:32.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836496 2024-09-13T15:49:32.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:49:32.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836496 2024-09-13T15:49:32.776 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836496 2024-09-13T15:49:32.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-09-13T15:49:32.776 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:49:33.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836495 -lt 21474836496 2024-09-13T15:49:33.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:49:34.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:49:34.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:49:34.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836496 -lt 21474836496 2024-09-13T15:49:34.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:49:34.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672972 2024-09-13T15:49:34.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:49:34.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:49:34.449 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672972 2024-09-13T15:49:34.449 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:49:34.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672972 2024-09-13T15:49:34.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-13T15:49:34.451 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672972 2024-09-13T15:49:34.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:49:34.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-13T15:49:34.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:49:34.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:49:34.819 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:49:35.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:49:35.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:49:35.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:49:35.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:49:35.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:49:35.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:49:35.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:49:35.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:49:35.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:49:35.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:49:35.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:49:35.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:49:35.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:49:35.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:49:35.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:49:35.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: ceph osd dump 2024-09-13T15:49:35.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: awk '{ print $2 }' 2024-09-13T15:49:35.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2024-09-13T15:49:36.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: poolid=1 2024-09-13T15:49:36.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:756: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.471165 bs=1032 count=1 2024-09-13T15:49:36.359 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T15:49:36.359 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T15:49:36.359 INFO:tasks.workunit.client.0.smithi112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000116244 s, 8.9 MB/s 2024-09-13T15:49:36.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 1 16 2024-09-13T15:49:36.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:36.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj1 testdata.471165 2024-09-13T15:49:36.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:36.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj2 testdata.471165 2024-09-13T15:49:36.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:36.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj3 testdata.471165 2024-09-13T15:49:37.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:37.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj4 testdata.471165 2024-09-13T15:49:37.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:37.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj5 testdata.471165 2024-09-13T15:49:37.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:37.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj6 testdata.471165 2024-09-13T15:49:38.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:38.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj7 testdata.471165 2024-09-13T15:49:38.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:38.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj8 testdata.471165 2024-09-13T15:49:38.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:38.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj9 testdata.471165 2024-09-13T15:49:38.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:38.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj10 testdata.471165 2024-09-13T15:49:39.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:39.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj11 testdata.471165 2024-09-13T15:49:39.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:39.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj12 testdata.471165 2024-09-13T15:49:39.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:39.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj13 testdata.471165 2024-09-13T15:49:39.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:39.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj14 testdata.471165 2024-09-13T15:49:39.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:39.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj15 testdata.471165 2024-09-13T15:49:40.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:49:40.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj16 testdata.471165 2024-09-13T15:49:40.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: get_primary test obj1 2024-09-13T15:49:40.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-13T15:49:40.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T15:49:40.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:49:40.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-13T15:49:40.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: local primary=1 2024-09-13T15:49:40.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: get_not_primary test obj1 2024-09-13T15:49:40.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2024-09-13T15:49:40.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-13T15:49:40.615 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2024-09-13T15:49:40.615 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-13T15:49:40.615 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T15:49:40.616 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-13T15:49:40.616 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:49:40.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T15:49:40.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2024-09-13T15:49:40.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T15:49:41.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: local replica=0 2024-09-13T15:49:41.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.471165 '$primary' 2024-09-13T15:49:41.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.471165 1 2024-09-13T15:49:41.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-09-13T15:49:41.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-09-13T15:49:41.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.471165 2024-09-13T15:49:41.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=1 2024-09-13T15:49:41.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-09-13T15:49:41.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-09-13T15:49:41.647 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap1 2024-09-13T15:49:41.651 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=1 2024-09-13T15:49:41.653 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T15:49:41.653 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T15:49:41.653 INFO:tasks.workunit.client.0.smithi112.stderr:256 bytes copied, 0.000258904 s, 989 kB/s 2024-09-13T15:49:41.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.471165 2024-09-13T15:49:42.058 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.471165 2024-09-13T15:49:42.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.471165 2024-09-13T15:49:42.559 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-09-13T15:49:42.560 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:49:42.560 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.471165 2024-09-13T15:49:42.785 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:49:42.785 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.471165 2024-09-13T15:49:42.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:49:42.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.471165 2024-09-13T15:49:43.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:49:43.178 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.471165 2024-09-13T15:49:43.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:49:43.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.471165 2024-09-13T15:49:43.587 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:49:43.587 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.471165 2024-09-13T15:49:43.796 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:49:43.796 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.471165 2024-09-13T15:49:43.996 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:49:43.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.471165 2024-09-13T15:49:44.223 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:49:44.223 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.471165 2024-09-13T15:49:44.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-09-13T15:49:44.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-09-13T15:49:44.732 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap2 2024-09-13T15:49:44.735 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=2 2024-09-13T15:49:44.737 INFO:tasks.workunit.client.0.smithi112.stderr:2+0 records in 2024-09-13T15:49:44.737 INFO:tasks.workunit.client.0.smithi112.stderr:2+0 records out 2024-09-13T15:49:44.737 INFO:tasks.workunit.client.0.smithi112.stderr:512 bytes copied, 0.000265301 s, 1.9 MB/s 2024-09-13T15:49:44.737 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.471165 2024-09-13T15:49:45.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-09-13T15:49:45.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-09-13T15:49:45.351 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap3 2024-09-13T15:49:45.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=3 2024-09-13T15:49:45.356 INFO:tasks.workunit.client.0.smithi112.stderr:3+0 records in 2024-09-13T15:49:45.356 INFO:tasks.workunit.client.0.smithi112.stderr:3+0 records out 2024-09-13T15:49:45.356 INFO:tasks.workunit.client.0.smithi112.stderr:768 bytes copied, 0.000277534 s, 2.8 MB/s 2024-09-13T15:49:45.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.471165 2024-09-13T15:49:45.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-09-13T15:49:45.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-09-13T15:49:45.969 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap4 2024-09-13T15:49:45.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=4 2024-09-13T15:49:45.975 INFO:tasks.workunit.client.0.smithi112.stderr:4+0 records in 2024-09-13T15:49:45.975 INFO:tasks.workunit.client.0.smithi112.stderr:4+0 records out 2024-09-13T15:49:45.975 INFO:tasks.workunit.client.0.smithi112.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000169198 s, 6.1 MB/s 2024-09-13T15:49:45.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.471165 2024-09-13T15:49:46.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.471165 2024-09-13T15:49:46.555 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-09-13T15:49:46.555 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-09-13T15:49:46.822 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap5 2024-09-13T15:49:46.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-09-13T15:49:46.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-09-13T15:49:47.103 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap6 2024-09-13T15:49:47.107 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=6 2024-09-13T15:49:47.108 INFO:tasks.workunit.client.0.smithi112.stderr:6+0 records in 2024-09-13T15:49:47.109 INFO:tasks.workunit.client.0.smithi112.stderr:6+0 records out 2024-09-13T15:49:47.109 INFO:tasks.workunit.client.0.smithi112.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 0.000292753 s, 5.2 MB/s 2024-09-13T15:49:47.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.471165 2024-09-13T15:49:47.432 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-09-13T15:49:47.432 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-09-13T15:49:47.700 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap7 2024-09-13T15:49:47.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-09-13T15:49:48.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-09-13T15:49:48.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-09-13T15:49:48.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-09-13T15:49:48.437 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:49:48.437 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:49:48.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:49:48.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:49:48.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:49:49.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:49:49.750 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj1 2024-09-13T15:49:50.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:49:50.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-09-13T15:49:52.814 INFO:tasks.workunit.client.0.smithi112.stdout:WARNING: only removing head with clones present 2024-09-13T15:49:52.814 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-13T15:49:53.349 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-09-13T15:49:53.349 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-09-13T15:49:54.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:49:54.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-13T15:49:55.418 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:c52c9666:::obj5:2# 2024-09-13T15:49:55.962 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-09-13T15:49:55.962 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-09-13T15:49:56.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:49:56.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:49:56.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-13T15:49:57.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-13T15:49:57.968 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:49:57.968 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-13T15:49:57.968 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:49:57.968 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:49:57.969 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-13T15:49:57.969 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-13T15:49:57.969 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-13T15:49:57.969 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-13T15:49:57.969 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-13T15:49:57.969 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-13T15:49:57.969 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:49:57.969 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-13T15:49:57.969 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-13T15:49:57.969 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-13T15:49:57.969 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-13T15:49:57.969 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:49:57.969 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:49:57.970 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:49:57.970 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:49:57.970 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:49:57.970 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:49:57.970 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:49:57.970 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:49:57.970 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:49:57.970 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:49:57.970 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:49:57.970 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:49:57.970 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:49:57.970 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:49:57.970 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:49:57.971 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:49:57.971 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:49:57.971 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:49:57.971 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:49:57.971 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:49:57.971 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:49:57.971 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:49:57.971 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:49:57.971 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:49:57.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-13T15:49:57.972 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:49:57.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-13T15:49:59.580 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:c52c9666:::obj5:1# 2024-09-13T15:50:00.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-13T15:50:01.224 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-13T15:50:01.226 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:50:01.226 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-13T15:50:01.226 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:01.226 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:50:01.226 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-13T15:50:01.226 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-13T15:50:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-13T15:50:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-13T15:50:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-13T15:50:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-13T15:50:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:50:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-13T15:50:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-13T15:50:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-13T15:50:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-13T15:50:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:50:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:50:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:50:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:50:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:50:01.228 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:01.228 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:50:01.228 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:50:01.228 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:50:01.228 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:01.228 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:50:01.228 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:50:01.228 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:50:01.228 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:01.228 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:50:01.228 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:50:01.228 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:50:01.228 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:01.229 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:50:01.229 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:50:01.229 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:50:01.229 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:01.229 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:50:01.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-13T15:50:01.229 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:50:01.230 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-13T15:50:01.230 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-09-13T15:50:01.231 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-09-13T15:50:02.156 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:02.156 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=18 2024-09-13T15:50:02.157 INFO:tasks.workunit.client.0.smithi112.stderr:18+0 records in 2024-09-13T15:50:02.157 INFO:tasks.workunit.client.0.smithi112.stderr:18+0 records out 2024-09-13T15:50:02.158 INFO:tasks.workunit.client.0.smithi112.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000292675 s, 15.7 MB/s 2024-09-13T15:50:02.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.471165 2024-09-13T15:50:04.025 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj3 2024-09-13T15:50:04.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:04.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=15 2024-09-13T15:50:04.930 INFO:tasks.workunit.client.0.smithi112.stderr:15+0 records in 2024-09-13T15:50:04.930 INFO:tasks.workunit.client.0.smithi112.stderr:15+0 records out 2024-09-13T15:50:04.931 INFO:tasks.workunit.client.0.smithi112.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000392588 s, 9.8 MB/s 2024-09-13T15:50:04.931 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.471165 2024-09-13T15:50:06.914 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj4 2024-09-13T15:50:06.914 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-09-13T15:50:07.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:07.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-13T15:50:09.323 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:0ee9ae15:::obj4:7# 2024-09-13T15:50:10.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-13T15:50:11.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-13T15:50:11.473 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-13T15:50:11.473 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:11.473 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:50:11.473 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-13T15:50:11.473 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-13T15:50:11.473 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-13T15:50:11.473 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-13T15:50:11.474 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-13T15:50:11.474 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-13T15:50:11.474 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:50:11.474 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-13T15:50:11.474 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-13T15:50:11.474 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-13T15:50:11.474 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-13T15:50:11.474 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:50:11.474 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:11.474 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:50:11.474 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:50:11.474 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:11.474 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:50:11.475 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:50:11.475 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:11.475 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:50:11.475 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:50:11.475 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:11.475 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:50:11.475 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:50:11.475 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:11.475 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:50:11.475 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:50:11.475 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:11.475 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:50:11.476 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-13T15:50:11.476 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:11.476 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:11.476 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:11.476 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:11.476 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:11.476 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:11.476 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:50:11.477 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj16 2024-09-13T15:50:11.477 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-09-13T15:50:12.365 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:12.366 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-13T15:50:13.651 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:461f8b5e:::obj16:7# 2024-09-13T15:50:14.187 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-09-13T15:50:15.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-13T15:50:15.294 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-13T15:50:15.294 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:50:15.294 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-13T15:50:15.294 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-13T15:50:15.294 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-13T15:50:15.294 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-13T15:50:15.294 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-13T15:50:15.295 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-13T15:50:15.295 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:50:15.295 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-13T15:50:15.295 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-13T15:50:15.295 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-13T15:50:15.295 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-13T15:50:15.295 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:50:15.295 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:50:15.295 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:50:15.295 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:50:15.295 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:50:15.295 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:50:15.296 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:50:15.296 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:50:15.296 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:50:15.296 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:50:15.296 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:50:15.296 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:50:15.296 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-13T15:50:15.296 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-13T15:50:15.297 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj2 2024-09-13T15:50:16.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:16.205 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-09-13T15:50:17.964 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:17.964 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-09-13T15:50:17.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:17.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=7 2024-09-13T15:50:17.968 INFO:tasks.workunit.client.0.smithi112.stderr:7+0 records in 2024-09-13T15:50:17.968 INFO:tasks.workunit.client.0.smithi112.stderr:7+0 records out 2024-09-13T15:50:17.968 INFO:tasks.workunit.client.0.smithi112.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 0.000188029 s, 9.5 MB/s 2024-09-13T15:50:17.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.471165 2024-09-13T15:50:19.881 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj6 2024-09-13T15:50:20.293 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:50:20.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:20.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-09-13T15:50:22.413 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj7 2024-09-13T15:50:22.787 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:50:23.325 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:23.325 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-09-13T15:50:25.782 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj8 2024-09-13T15:50:26.147 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:50:26.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:26.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-09-13T15:50:28.236 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj9 2024-09-13T15:50:28.672 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:50:28.956 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:28.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-09-13T15:50:30.674 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj10 2024-09-13T15:50:31.030 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:50:31.564 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:31.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-09-13T15:50:33.966 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj11 2024-09-13T15:50:34.338 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:50:34.628 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:34.628 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-09-13T15:50:36.361 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj12 2024-09-13T15:50:36.730 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:50:37.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:37.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-09-13T15:50:38.829 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj13 2024-09-13T15:50:39.197 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:50:39.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:39.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-09-13T15:50:41.576 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj14 2024-09-13T15:50:41.947 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:50:42.240 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:42.240 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-09-13T15:50:44.469 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-09-13T15:50:44.470 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj15 2024-09-13T15:50:44.843 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:50:45.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:50:45.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-09-13T15:50:47.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-09-13T15:50:47.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-09-13T15:50:47.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: rm -f testdata.471165 2024-09-13T15:50:47.335 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: expr 2 - 1 2024-09-13T15:50:47.336 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: seq 0 1 2024-09-13T15:50:47.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:50:47.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2024-09-13T15:50:47.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2024-09-13T15:50:47.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:50:47.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:50:47.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:50:47.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-13T15:50:47.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T15:50:47.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:50:47.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:50:47.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:50:47.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-13T15:50:47.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-13T15:50:47.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:50:47.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:50:47.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-13T15:50:47.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:50:47.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:50:47.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:50:47.342 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:50:47.342 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:50:47.342 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:50:47.342 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:50:47.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:50:47.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:50:47.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-13T15:50:47.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-13T15:50:47.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:50:47.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:50:47.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:50:47.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:50:47.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:50:47.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:50:47.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-13T15:50:47.343 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:50:47.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:50:47.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:50:47.344 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-09-13T15:50:47.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:50:47.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:50:47.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:50:47.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:50:47.389 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:50:47.386+0000 7fd5d36637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:50:47.391 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:50:47.390+0000 7fd5d36637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:50:47.392 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:50:47.390+0000 7fd5d36637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:50:47.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:50:47.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:50:47.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:50:47.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:50:47.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:50:47.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:50:47.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:50:47.695 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:50:47.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:50:47.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:50:48.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:50:48.673 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:50:48.670+0000 7fd5d36637c0 -1 Falling back to public interface 2024-09-13T15:50:49.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:50:49.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:50:49.076 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:50:49.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:50:49.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:50:49.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:50:49.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:50:50.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:50:50.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:50:50.433 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:50:50.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:50:50.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:50:50.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:50:50.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:50:51.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:50:51.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:50:51.813 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:50:51.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:50:51.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:50:51.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:50:52.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:50:52.434 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:50:52.430+0000 7fd5d36637c0 -1 osd.0 23 log_to_monitors true 2024-09-13T15:50:53.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:50:53.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:50:53.166 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:50:53.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:50:53.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:50:53.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:50:53.564 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/4257028248,v1:127.0.0.1:6803/4257028248] [v2:127.0.0.1:6804/4257028248,v1:127.0.0.1:6805/4257028248] exists,up ff85374b-00bd-419c-a965-df40eb1e79c7 2024-09-13T15:50:53.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:50:53.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:50:53.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:50:53.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:50:53.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2024-09-13T15:50:53.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2024-09-13T15:50:53.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:50:53.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:50:53.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:50:53.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-13T15:50:53.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T15:50:53.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:50:53.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:50:53.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:50:53.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-13T15:50:53.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-13T15:50:53.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:50:53.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:50:53.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-13T15:50:53.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:50:53.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:50:53.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:50:53.569 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:50:53.569 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:50:53.569 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:50:53.569 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:50:53.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:50:53.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:50:53.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-13T15:50:53.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-13T15:50:53.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:50:53.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:50:53.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:50:53.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:50:53.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:50:53.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:50:53.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-13T15:50:53.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:50:53.571 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:50:53.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:50:53.572 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/1/whoami 2024-09-13T15:50:53.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:50:53.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:50:53.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:50:53.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:50:53.613 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:50:53.610+0000 7fa5eb53f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:50:53.660 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:50:53.658+0000 7fa5eb53f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:50:53.661 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:50:53.658+0000 7fa5eb53f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:50:53.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:50:53.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:50:53.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:50:53.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:50:53.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:50:53.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:50:53.894 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:50:53.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:50:53.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:50:53.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:50:54.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:50:54.737 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:50:54.734+0000 7fa5eb53f7c0 -1 Falling back to public interface 2024-09-13T15:50:55.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:50:55.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:50:55.236 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:50:55.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:50:55.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:50:55.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:50:55.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:50:56.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:50:56.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:50:56.589 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:50:56.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:50:56.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:50:56.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:50:56.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:50:57.667 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:50:57.666+0000 7fa5eb53f7c0 -1 osd.1 23 log_to_monitors true 2024-09-13T15:50:57.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:50:57.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:50:57.925 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:50:57.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:50:57.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:50:57.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:50:58.308 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 29 up_thru 14 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/542873127,v1:127.0.0.1:6811/542873127] [v2:127.0.0.1:6812/542873127,v1:127.0.0.1:6813/542873127] exists,up c3dd738e-e589-48a3-b2dd-5916f6bdb2a9 2024-09-13T15:50:58.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:50:58.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:50:58.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:50:58.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2024-09-13T15:50:58.457 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T15:50:58.457 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-13T15:50:58.458 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:50:58.471 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T15:50:58.471 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-13T15:50:58.471 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:50:58.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2024-09-13T15:50:58.741 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T15:50:58.741 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-13T15:50:58.741 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:50:58.753 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T15:50:58.753 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-13T15:50:58.753 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:50:58.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2024-09-13T15:50:58.947 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T15:50:58.947 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-13T15:50:58.947 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:50:58.961 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T15:50:58.962 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-13T15:50:58.962 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:50:58.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:777: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2024-09-13T15:50:59.222 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T15:50:59.222 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-09-13T15:50:59.222 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:50:59.236 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T15:50:59.236 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-09-13T15:50:59.236 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:50:59.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2024-09-13T15:50:59.399 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T15:50:59.400 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-09-13T15:50:59.400 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:50:59.414 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T15:50:59.414 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-09-13T15:50:59.414 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:50:59.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: wait_for_clean 2024-09-13T15:50:59.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:50:59.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:50:59.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:50:59.430 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:50:59.431 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:50:59.431 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:50:59.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:50:59.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:50:59.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:50:59.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:50:59.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:50:59.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:50:59.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:50:59.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:50:59.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:50:59.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:50:59.846 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:50:59.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:50:59.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:50:59.846 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:51:00.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149699 2024-09-13T15:51:00.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149699 2024-09-13T15:51:00.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149699' 2024-09-13T15:51:00.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:51:00.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:51:00.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051586 2024-09-13T15:51:00.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051586 2024-09-13T15:51:00.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149699 1-124554051586' 2024-09-13T15:51:00.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:51:00.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149699 2024-09-13T15:51:00.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:51:00.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:51:00.186 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149699 2024-09-13T15:51:00.187 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:51:00.187 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 111669149699 2024-09-13T15:51:00.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149699 2024-09-13T15:51:00.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149699' 2024-09-13T15:51:00.188 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:51:00.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149699 -lt 111669149699 2024-09-13T15:51:00.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:51:00.549 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-124554051586 2024-09-13T15:51:00.549 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:51:00.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:51:00.552 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-124554051586 2024-09-13T15:51:00.552 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:51:00.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051586 2024-09-13T15:51:00.554 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 124554051586 2024-09-13T15:51:00.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-09-13T15:51:00.554 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:51:00.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 124554051586 2024-09-13T15:51:00.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:51:01.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:51:01.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:51:02.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 124554051586 2024-09-13T15:51:02.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:51:03.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T15:51:03.283 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:51:03.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-13T15:51:03.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:51:03.621 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:51:03.621 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:51:04.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:51:04.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:51:04.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:51:04.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:51:04.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:51:04.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:51:04.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:51:04.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:51:04.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:51:04.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:51:04.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:51:04.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:51:04.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:51:04.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:51:04.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:51:04.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:781: _scrub_snaps_multi: local pgid=1.0 2024-09-13T15:51:04.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:782: _scrub_snaps_multi: pg_scrub 1.0 2024-09-13T15:51:04.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-13T15:51:04.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-13T15:51:04.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:51:04.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:51:04.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:51:04.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:51:05.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-13T15:49:30.766364+0000 2024-09-13T15:51:05.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-13T15:51:05.564 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.0 on osd.1 to scrub 2024-09-13T15:51:05.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-13T15:49:30.766364+0000 2024-09-13T15:51:05.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T15:51:05.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:49:30.766364+0000 2024-09-13T15:51:05.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:51:05.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:51:05.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:51:05.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:51:05.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:51:05.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:51:05.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:51:05.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:51:05.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:49:30.766364+0000 '>' 2024-09-13T15:49:30.766364+0000 2024-09-13T15:51:05.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:51:06.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:51:06.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:51:06.947 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:51:06.947 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:51:06.947 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:51:06.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:51:06.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:51:07.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:49:30.766364+0000 '>' 2024-09-13T15:49:30.766364+0000 2024-09-13T15:51:07.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:51:08.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:51:08.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:51:08.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:51:08.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:51:08.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:51:08.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:51:08.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:51:08.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:51:06.461972+0000 '>' 2024-09-13T15:49:30.766364+0000 2024-09-13T15:51:08.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:51:08.649 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:51:08.649 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: wc -l 2024-09-13T15:51:08.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: test 14 -gt 3 2024-09-13T15:51:08.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:51:08.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: wc -l 2024-09-13T15:51:08.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: test 14 -gt 3 2024-09-13T15:51:08.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:789: _scrub_snaps_multi: rados list-inconsistent-pg test 2024-09-13T15:51:08.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2024-09-13T15:51:08.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: test 1 = 1 2024-09-13T15:51:08.731 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2024-09-13T15:51:08.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: test 1.0 = 1.0 2024-09-13T15:51:08.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2024-09-13T15:51:08.828 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:51:08.828 INFO:tasks.workunit.client.0.smithi112.stdout: "epoch": 29, 2024-09-13T15:51:08.828 INFO:tasks.workunit.client.0.smithi112.stdout: "inconsistents": [ 2024-09-13T15:51:08.828 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.829 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.829 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj5", 2024-09-13T15:51:08.829 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.829 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.829 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:51:08.829 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 5 2024-09-13T15:51:08.829 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.829 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.829 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:51:08.829 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:51:08.829 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.829 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.829 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.829 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj5", 2024-09-13T15:51:08.830 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.830 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": 1, 2024-09-13T15:51:08.830 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1718170787, 2024-09-13T15:51:08.830 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.830 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.830 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.830 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.830 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'19", 2024-09-13T15:51:08.830 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'5", 2024-09-13T15:51:08.830 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4265.0:1", 2024-09-13T15:51:08.830 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 5, 2024-09-13T15:51:08.830 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.831 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:37.484429+0000", 2024-09-13T15:51:08.831 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:37.485239+0000", 2024-09-13T15:51:08.831 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.831 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.831 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.831 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.831 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.831 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.831 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.831 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0xbcdc0a93", 2024-09-13T15:51:08.831 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.831 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.831 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.831 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.832 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.832 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.832 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.832 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.832 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.832 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.832 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.832 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.832 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.832 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.833 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032 2024-09-13T15:51:08.833 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.833 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.833 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.833 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.833 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.833 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:51:08.833 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.833 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.833 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.833 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.833 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.833 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.834 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj5", 2024-09-13T15:51:08.834 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.834 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.834 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 2, 2024-09-13T15:51:08.834 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 20 2024-09-13T15:51:08.834 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.834 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.834 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:51:08.834 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:51:08.834 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.834 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.834 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.834 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj5", 2024-09-13T15:51:08.834 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.835 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": 2, 2024-09-13T15:51:08.835 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1718170787, 2024-09-13T15:51:08.835 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.835 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.835 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.835 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.835 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "18'41", 2024-09-13T15:51:08.835 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "17'20", 2024-09-13T15:51:08.835 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4313.0:1", 2024-09-13T15:51:08.835 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 20, 2024-09-13T15:51:08.835 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.835 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:42.104071+0000", 2024-09-13T15:51:08.835 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:42.104808+0000", 2024-09-13T15:51:08.836 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.836 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.836 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.836 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.836 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.836 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.836 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.836 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x337e9189", 2024-09-13T15:51:08.836 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.836 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.836 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.836 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.836 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.837 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.837 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.837 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.837 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.837 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.837 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.837 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.837 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.837 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.837 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256 2024-09-13T15:51:08.837 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.837 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.837 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.837 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.838 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.838 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:51:08.838 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.838 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.838 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.838 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.838 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.838 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.838 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj5", 2024-09-13T15:51:08.838 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.838 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.838 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 4, 2024-09-13T15:51:08.838 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 42 2024-09-13T15:51:08.838 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.839 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.839 INFO:tasks.workunit.client.0.smithi112.stdout: "size_mismatch" 2024-09-13T15:51:08.839 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.839 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:51:08.839 INFO:tasks.workunit.client.0.smithi112.stdout: "size_mismatch_info", 2024-09-13T15:51:08.839 INFO:tasks.workunit.client.0.smithi112.stdout: "obj_size_info_mismatch" 2024-09-13T15:51:08.839 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.839 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.839 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.839 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj5", 2024-09-13T15:51:08.839 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.839 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": 4, 2024-09-13T15:51:08.839 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1718170787, 2024-09-13T15:51:08.839 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.840 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.840 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.840 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.840 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "20'45", 2024-09-13T15:51:08.840 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "18'42", 2024-09-13T15:51:08.840 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4349.0:1", 2024-09-13T15:51:08.840 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 42, 2024-09-13T15:51:08.840 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 512, 2024-09-13T15:51:08.840 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:44.781186+0000", 2024-09-13T15:51:08.840 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:44.897872+0000", 2024-09-13T15:51:08.840 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.840 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.840 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.841 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.841 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.841 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.841 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.841 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x30b951b5", 2024-09-13T15:51:08.841 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.841 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.841 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.841 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.841 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.842 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.842 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.842 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.842 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.842 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.842 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.842 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.842 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.842 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.842 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 512 2024-09-13T15:51:08.842 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.842 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.842 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.842 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.843 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.843 INFO:tasks.workunit.client.0.smithi112.stdout: "size_mismatch_info", 2024-09-13T15:51:08.843 INFO:tasks.workunit.client.0.smithi112.stdout: "obj_size_info_mismatch" 2024-09-13T15:51:08.843 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.843 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 4608, 2024-09-13T15:51:08.843 INFO:tasks.workunit.client.0.smithi112.stdout: "object_info": { 2024-09-13T15:51:08.843 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.843 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj5", 2024-09-13T15:51:08.843 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.843 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": 4, 2024-09-13T15:51:08.844 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1718170787, 2024-09-13T15:51:08.844 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.844 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.844 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.844 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.844 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "20'45", 2024-09-13T15:51:08.844 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "18'42", 2024-09-13T15:51:08.844 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4349.0:1", 2024-09-13T15:51:08.844 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 42, 2024-09-13T15:51:08.844 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 512, 2024-09-13T15:51:08.844 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:44.781186+0000", 2024-09-13T15:51:08.845 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:44.897872+0000", 2024-09-13T15:51:08.845 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.845 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.845 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.845 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.845 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.845 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.845 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.845 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x30b951b5", 2024-09-13T15:51:08.845 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.846 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.846 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.846 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.846 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.846 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.846 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.846 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.846 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.846 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.846 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.846 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.846 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.846 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.846 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj4", 2024-09-13T15:51:08.847 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.847 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.847 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 7, 2024-09-13T15:51:08.847 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 4 2024-09-13T15:51:08.847 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.847 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.847 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:51:08.847 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:51:08.847 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.847 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.847 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.847 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj4", 2024-09-13T15:51:08.847 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.847 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": 7, 2024-09-13T15:51:08.848 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 2826278768, 2024-09-13T15:51:08.848 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.848 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.848 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.848 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.848 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "23'51", 2024-09-13T15:51:08.848 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'4", 2024-09-13T15:51:08.848 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4262.0:1", 2024-09-13T15:51:08.848 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 4, 2024-09-13T15:51:08.848 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.848 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:37.268284+0000", 2024-09-13T15:51:08.848 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:37.269087+0000", 2024-09-13T15:51:08.848 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.849 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.849 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.849 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.849 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.849 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.849 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.849 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0xbcdc0a93", 2024-09-13T15:51:08.849 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.849 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.849 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.850 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.850 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.850 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.850 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.850 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.850 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.850 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.850 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.850 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.850 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.850 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.850 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032 2024-09-13T15:51:08.850 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.850 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.851 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.851 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.851 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.851 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:51:08.851 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.851 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.851 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.851 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.851 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.851 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.851 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj5", 2024-09-13T15:51:08.851 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.851 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.851 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 7, 2024-09-13T15:51:08.852 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 0 2024-09-13T15:51:08.852 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.852 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.852 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:51:08.852 INFO:tasks.workunit.client.0.smithi112.stdout: "missing", 2024-09-13T15:51:08.852 INFO:tasks.workunit.client.0.smithi112.stdout: "info_missing" 2024-09-13T15:51:08.852 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.852 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.852 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.852 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.852 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.852 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.852 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:51:08.853 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.853 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.853 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.853 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.853 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.853 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.853 INFO:tasks.workunit.client.0.smithi112.stdout: "info_missing" 2024-09-13T15:51:08.853 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.853 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1792 2024-09-13T15:51:08.853 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.853 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.853 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.853 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.853 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.854 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj1", 2024-09-13T15:51:08.854 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.854 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.854 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:51:08.854 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 18 2024-09-13T15:51:08.854 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.854 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.854 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:51:08.854 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:51:08.854 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.854 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.854 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.855 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj1", 2024-09-13T15:51:08.855 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.855 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:51:08.855 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1828249343, 2024-09-13T15:51:08.855 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.855 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.855 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.855 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.855 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'18", 2024-09-13T15:51:08.855 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'1", 2024-09-13T15:51:08.855 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4310.0:1", 2024-09-13T15:51:08.855 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 18, 2024-09-13T15:51:08.855 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.856 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:41.698399+0000", 2024-09-13T15:51:08.856 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:41.821821+0000", 2024-09-13T15:51:08.856 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.856 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.856 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.856 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.856 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.856 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.856 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.856 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x337e9189", 2024-09-13T15:51:08.856 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.856 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.856 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.857 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.857 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.857 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.857 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.861 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.861 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.861 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.861 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.861 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.861 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.861 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.861 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256 2024-09-13T15:51:08.861 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.861 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.861 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.861 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.862 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.862 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:51:08.862 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.862 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.862 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.862 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.862 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.862 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.862 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj10", 2024-09-13T15:51:08.862 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.862 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.862 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:51:08.862 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 32 2024-09-13T15:51:08.862 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.863 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.863 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:51:08.863 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.863 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:51:08.863 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.863 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.863 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj10", 2024-09-13T15:51:08.863 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.863 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:51:08.863 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 718195851, 2024-09-13T15:51:08.863 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.863 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.863 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.863 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.864 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'32", 2024-09-13T15:51:08.864 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'10", 2024-09-13T15:51:08.864 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4331.0:1", 2024-09-13T15:51:08.864 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 32, 2024-09-13T15:51:08.864 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.864 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:43.425361+0000", 2024-09-13T15:51:08.864 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:43.426208+0000", 2024-09-13T15:51:08.864 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.864 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.864 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.864 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.864 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.864 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.865 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.865 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x337e9189", 2024-09-13T15:51:08.865 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.865 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.865 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.865 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.865 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.865 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.865 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.865 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.865 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.865 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.865 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.866 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.866 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.866 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.866 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.866 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.866 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:51:08.866 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:51:08.866 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.866 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:51:08.866 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.866 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:51:08.866 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:51:08.866 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:51:08.866 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.867 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.867 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.867 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.867 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.867 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.867 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.867 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.867 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.867 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.867 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.867 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:51:08.867 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:51:08.867 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.868 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:51:08.868 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.868 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "????", 2024-09-13T15:51:08.868 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:51:08.868 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:51:08.868 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.868 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.868 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.868 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.868 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.868 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.868 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.868 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.869 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.869 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj11", 2024-09-13T15:51:08.869 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.869 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.869 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:51:08.869 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 34 2024-09-13T15:51:08.869 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.869 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.869 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:51:08.869 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.869 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:51:08.869 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.869 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.870 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj11", 2024-09-13T15:51:08.870 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.870 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:51:08.870 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 693400951, 2024-09-13T15:51:08.870 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.870 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.870 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.870 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.870 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'34", 2024-09-13T15:51:08.870 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'11", 2024-09-13T15:51:08.870 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4334.0:1", 2024-09-13T15:51:08.870 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 34, 2024-09-13T15:51:08.870 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.870 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:43.635229+0000", 2024-09-13T15:51:08.871 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:43.635966+0000", 2024-09-13T15:51:08.871 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.871 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.871 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.871 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.871 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.871 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.871 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.871 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x337e9189", 2024-09-13T15:51:08.871 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.871 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.871 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.871 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.872 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.872 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.872 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.872 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.872 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.872 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.872 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.872 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.872 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.872 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.872 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.872 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.872 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:51:08.873 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:51:08.873 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.873 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:51:08.873 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.873 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:51:08.873 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:51:08.873 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:51:08.873 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.873 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.873 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.873 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.873 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.873 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.874 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.874 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.874 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.874 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.874 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.874 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:51:08.874 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [] 2024-09-13T15:51:08.874 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.874 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.874 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.874 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.874 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.874 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.875 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj13", 2024-09-13T15:51:08.875 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.875 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.875 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:51:08.875 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 38 2024-09-13T15:51:08.875 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.875 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.875 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:51:08.875 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.875 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:51:08.875 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.875 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.875 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj13", 2024-09-13T15:51:08.876 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.876 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:51:08.876 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 2087409765, 2024-09-13T15:51:08.876 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.876 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.876 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.876 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.876 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'38", 2024-09-13T15:51:08.876 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'13", 2024-09-13T15:51:08.876 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4340.0:1", 2024-09-13T15:51:08.876 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 38, 2024-09-13T15:51:08.876 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.876 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:44.041246+0000", 2024-09-13T15:51:08.877 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:44.042121+0000", 2024-09-13T15:51:08.877 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.877 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.877 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.877 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.877 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.877 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.877 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.877 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x337e9189", 2024-09-13T15:51:08.877 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.877 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.877 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.877 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.878 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.878 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.878 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.878 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.878 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.878 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.878 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.878 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.878 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.878 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.878 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.878 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.878 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:51:08.878 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:51:08.879 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.879 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:51:08.879 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.879 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:51:08.879 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:51:08.879 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:51:08.879 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.879 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.879 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.879 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.879 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.879 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.879 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.880 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.880 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.880 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.880 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.880 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:51:08.880 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:51:08.880 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.880 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:51:08.880 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.880 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]" 2024-09-13T15:51:08.880 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.880 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.880 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.880 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.881 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.881 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.881 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.881 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.881 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj14", 2024-09-13T15:51:08.881 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.881 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.881 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:51:08.881 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 40 2024-09-13T15:51:08.881 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.881 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.881 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:51:08.881 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.882 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:51:08.882 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.882 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.882 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj14", 2024-09-13T15:51:08.882 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.882 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:51:08.882 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 2484217095, 2024-09-13T15:51:08.882 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.882 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.882 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.882 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.882 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'40", 2024-09-13T15:51:08.882 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'14", 2024-09-13T15:51:08.882 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4343.0:1", 2024-09-13T15:51:08.883 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 40, 2024-09-13T15:51:08.883 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.883 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:44.271065+0000", 2024-09-13T15:51:08.883 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:44.271884+0000", 2024-09-13T15:51:08.883 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.883 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.883 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.883 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.883 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.883 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.883 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.883 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x337e9189", 2024-09-13T15:51:08.883 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.884 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.884 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.884 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.884 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.884 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.884 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.884 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.884 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.884 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.884 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.884 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.884 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.884 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.884 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.885 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.885 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:51:08.885 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:51:08.885 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.885 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:51:08.885 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.885 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:51:08.885 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:51:08.885 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:51:08.885 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.885 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.885 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.885 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.886 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.886 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.886 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.886 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.886 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.886 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.886 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.886 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:51:08.886 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:51:08.886 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.886 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:51:08.886 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1033, 2024-09-13T15:51:08.886 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:51:08.886 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:51:08.887 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:51:08.887 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.887 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.887 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.887 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.887 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.887 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.887 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.887 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.887 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.887 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj15", 2024-09-13T15:51:08.887 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.887 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.887 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:51:08.888 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 15 2024-09-13T15:51:08.888 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.888 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.888 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:51:08.888 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_corrupted" 2024-09-13T15:51:08.888 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.888 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.888 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.888 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj15", 2024-09-13T15:51:08.888 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.888 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:51:08.888 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 612772309, 2024-09-13T15:51:08.888 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.889 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.889 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.889 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.889 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "16'15", 2024-09-13T15:51:08.889 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "0'0", 2024-09-13T15:51:08.889 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4295.0:1", 2024-09-13T15:51:08.889 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 15, 2024-09-13T15:51:08.889 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.889 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:39.889855+0000", 2024-09-13T15:51:08.889 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:39.890575+0000", 2024-09-13T15:51:08.889 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.889 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.889 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.890 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.890 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.890 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.890 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.890 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0xbcdc0a93", 2024-09-13T15:51:08.890 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.890 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.890 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.890 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.890 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.890 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.890 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.890 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.891 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.891 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.891 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.891 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.891 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.891 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.891 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.891 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.891 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 0, 2024-09-13T15:51:08.891 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [] 2024-09-13T15:51:08.891 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.891 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.892 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.892 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.892 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.892 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.892 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_corrupted" 2024-09-13T15:51:08.892 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.892 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.892 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": "Z2FyYmFnZQo=" 2024-09-13T15:51:08.892 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.892 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.892 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.892 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.892 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.892 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj2", 2024-09-13T15:51:08.893 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.893 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.893 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:51:08.893 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 56 2024-09-13T15:51:08.893 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.893 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.893 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:51:08.893 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_missing" 2024-09-13T15:51:08.893 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.893 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.893 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.893 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj2", 2024-09-13T15:51:08.893 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.894 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:51:08.894 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1058988552, 2024-09-13T15:51:08.894 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.894 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.894 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.894 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.894 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "23'56", 2024-09-13T15:51:08.894 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "20'48", 2024-09-13T15:51:08.894 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4385.0:1", 2024-09-13T15:51:08.894 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 56, 2024-09-13T15:51:08.894 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 0, 2024-09-13T15:51:08.894 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:48.275773+0000", 2024-09-13T15:51:08.894 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:48.276534+0000", 2024-09-13T15:51:08.895 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.895 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.895 INFO:tasks.workunit.client.0.smithi112.stdout: "whiteout", 2024-09-13T15:51:08.895 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty" 2024-09-13T15:51:08.895 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.895 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.895 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.895 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0xffffffff", 2024-09-13T15:51:08.895 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.895 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.895 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.895 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.895 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.895 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.896 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.896 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.896 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.896 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.896 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.896 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.896 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.896 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.896 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 0, 2024-09-13T15:51:08.896 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.896 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 7, 2024-09-13T15:51:08.896 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:51:08.896 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.897 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 4, 2024-09-13T15:51:08.897 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.897 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:51:08.897 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:51:08.897 INFO:tasks.workunit.client.0.smithi112.stdout: 4, 2024-09-13T15:51:08.897 INFO:tasks.workunit.client.0.smithi112.stdout: 3, 2024-09-13T15:51:08.897 INFO:tasks.workunit.client.0.smithi112.stdout: 2, 2024-09-13T15:51:08.897 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:51:08.897 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.897 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.897 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.897 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 7, 2024-09-13T15:51:08.897 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1024, 2024-09-13T15:51:08.898 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:51:08.898 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:51:08.898 INFO:tasks.workunit.client.0.smithi112.stdout: 7, 2024-09-13T15:51:08.898 INFO:tasks.workunit.client.0.smithi112.stdout: 6, 2024-09-13T15:51:08.898 INFO:tasks.workunit.client.0.smithi112.stdout: 5 2024-09-13T15:51:08.898 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.898 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.898 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.898 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.898 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.898 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.898 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.898 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.899 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.899 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_missing" 2024-09-13T15:51:08.899 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.899 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 0 2024-09-13T15:51:08.899 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.899 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.899 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.899 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.899 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.899 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj3", 2024-09-13T15:51:08.899 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.899 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.899 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:51:08.899 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 44 2024-09-13T15:51:08.900 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.900 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.900 INFO:tasks.workunit.client.0.smithi112.stdout: "size_mismatch" 2024-09-13T15:51:08.900 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.900 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:51:08.900 INFO:tasks.workunit.client.0.smithi112.stdout: "size_mismatch_info", 2024-09-13T15:51:08.900 INFO:tasks.workunit.client.0.smithi112.stdout: "obj_size_info_mismatch" 2024-09-13T15:51:08.900 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.900 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.900 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.900 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj3", 2024-09-13T15:51:08.900 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.900 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:51:08.900 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1643547569, 2024-09-13T15:51:08.901 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.901 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.901 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.901 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.901 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "19'44", 2024-09-13T15:51:08.901 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "17'22", 2024-09-13T15:51:08.901 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4355.0:1", 2024-09-13T15:51:08.901 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 44, 2024-09-13T15:51:08.901 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 768, 2024-09-13T15:51:08.901 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:45.403414+0000", 2024-09-13T15:51:08.901 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:45.466365+0000", 2024-09-13T15:51:08.901 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.901 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.902 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.902 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.902 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.902 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.902 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.902 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x98093861", 2024-09-13T15:51:08.902 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.902 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.902 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.902 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.902 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.902 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.902 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.902 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.903 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.903 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.903 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.903 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.903 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.903 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.903 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 768 2024-09-13T15:51:08.903 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.903 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.903 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.903 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.903 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.903 INFO:tasks.workunit.client.0.smithi112.stdout: "size_mismatch_info", 2024-09-13T15:51:08.904 INFO:tasks.workunit.client.0.smithi112.stdout: "obj_size_info_mismatch" 2024-09-13T15:51:08.904 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.904 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 3840, 2024-09-13T15:51:08.904 INFO:tasks.workunit.client.0.smithi112.stdout: "object_info": { 2024-09-13T15:51:08.904 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.904 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj3", 2024-09-13T15:51:08.904 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.904 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:51:08.904 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1643547569, 2024-09-13T15:51:08.904 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.904 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.904 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.904 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.905 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "19'44", 2024-09-13T15:51:08.905 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "17'22", 2024-09-13T15:51:08.905 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4355.0:1", 2024-09-13T15:51:08.905 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 44, 2024-09-13T15:51:08.905 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 768, 2024-09-13T15:51:08.905 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:45.403414+0000", 2024-09-13T15:51:08.905 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:45.466365+0000", 2024-09-13T15:51:08.905 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.905 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.905 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.905 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.905 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.905 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.905 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.906 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x98093861", 2024-09-13T15:51:08.906 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.906 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.906 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.906 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.906 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.906 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.906 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.906 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.906 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.906 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.906 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.906 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.906 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.907 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.907 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj6", 2024-09-13T15:51:08.907 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.907 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.907 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:51:08.907 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 24 2024-09-13T15:51:08.907 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.907 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.907 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:51:08.907 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.907 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:51:08.907 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.908 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.908 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj6", 2024-09-13T15:51:08.908 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.908 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:51:08.908 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 2202164420, 2024-09-13T15:51:08.908 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.908 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.908 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.908 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.908 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'24", 2024-09-13T15:51:08.908 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'6", 2024-09-13T15:51:08.908 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4319.0:1", 2024-09-13T15:51:08.908 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 24, 2024-09-13T15:51:08.909 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.909 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:42.622448+0000", 2024-09-13T15:51:08.909 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:42.623192+0000", 2024-09-13T15:51:08.909 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.909 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.909 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.909 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.909 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.909 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.909 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.909 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x337e9189", 2024-09-13T15:51:08.909 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.909 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.909 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.910 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.910 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.910 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.910 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.910 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.911 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.911 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.911 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.911 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.911 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.911 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.911 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.911 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.911 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:51:08.911 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:51:08.911 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.911 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:51:08.911 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.912 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:51:08.912 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:51:08.912 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:51:08.912 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.912 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.912 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.912 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.912 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.912 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.912 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.912 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.912 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.912 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.912 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.913 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:51:08.913 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [] 2024-09-13T15:51:08.913 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.913 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.913 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.913 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.913 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.913 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.913 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj7", 2024-09-13T15:51:08.913 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.913 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.913 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:51:08.913 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 26 2024-09-13T15:51:08.914 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.914 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.914 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:51:08.914 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.914 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:51:08.914 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.914 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.914 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj7", 2024-09-13T15:51:08.914 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.914 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:51:08.914 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1552453721, 2024-09-13T15:51:08.914 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.914 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.914 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.915 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.915 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'26", 2024-09-13T15:51:08.915 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'7", 2024-09-13T15:51:08.915 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4322.0:1", 2024-09-13T15:51:08.915 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 26, 2024-09-13T15:51:08.915 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.915 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:42.832329+0000", 2024-09-13T15:51:08.915 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:42.833122+0000", 2024-09-13T15:51:08.915 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.915 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.915 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.915 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.915 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.916 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.916 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.916 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x337e9189", 2024-09-13T15:51:08.916 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.916 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.916 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.916 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.916 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.916 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.916 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.916 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.916 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.916 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.917 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.917 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.917 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.917 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.917 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.917 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.917 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:51:08.917 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:51:08.917 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.917 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:51:08.917 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.917 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:51:08.917 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:51:08.918 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:51:08.918 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.918 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.918 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.918 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.918 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.918 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.918 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.918 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.918 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.918 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.918 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.918 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 0, 2024-09-13T15:51:08.919 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [] 2024-09-13T15:51:08.919 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.919 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.919 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.919 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.919 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.919 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.919 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj8", 2024-09-13T15:51:08.919 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.919 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.919 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:51:08.919 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 28 2024-09-13T15:51:08.919 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.919 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.920 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:51:08.920 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.920 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:51:08.920 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.920 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.920 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj8", 2024-09-13T15:51:08.920 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.920 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:51:08.920 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 2381834917, 2024-09-13T15:51:08.920 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.920 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.920 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.920 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.920 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'28", 2024-09-13T15:51:08.921 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'8", 2024-09-13T15:51:08.921 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4325.0:1", 2024-09-13T15:51:08.921 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 28, 2024-09-13T15:51:08.921 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.921 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:43.031825+0000", 2024-09-13T15:51:08.921 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:43.032548+0000", 2024-09-13T15:51:08.921 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.921 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.921 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.921 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.921 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.921 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.921 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.922 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x337e9189", 2024-09-13T15:51:08.922 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.922 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.922 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.922 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.922 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.922 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.922 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.922 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.922 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.922 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.922 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.922 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.922 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.923 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.923 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.923 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.923 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:51:08.923 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:51:08.923 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.923 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:51:08.923 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.923 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:51:08.923 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:51:08.923 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:51:08.923 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.923 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.924 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.924 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.924 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.924 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.924 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.924 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.924 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.924 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.924 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.924 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 0, 2024-09-13T15:51:08.924 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:51:08.924 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.924 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:51:08.925 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.925 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:51:08.925 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:51:08.925 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:51:08.925 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.925 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.925 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.925 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.925 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.925 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.925 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.925 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.925 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:51:08.925 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj9", 2024-09-13T15:51:08.926 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:51:08.926 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:51:08.926 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:51:08.926 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 30 2024-09-13T15:51:08.926 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.926 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:51:08.926 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:51:08.926 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.926 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:51:08.926 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:51:08.926 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:51:08.926 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj9", 2024-09-13T15:51:08.926 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:51:08.927 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:51:08.927 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 3833113727, 2024-09-13T15:51:08.927 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:51:08.927 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:51:08.927 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:51:08.927 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.927 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'30", 2024-09-13T15:51:08.927 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'9", 2024-09-13T15:51:08.927 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4328.0:1", 2024-09-13T15:51:08.927 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 30, 2024-09-13T15:51:08.927 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.927 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:49:43.224959+0000", 2024-09-13T15:51:08.927 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:49:43.225666+0000", 2024-09-13T15:51:08.928 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:51:08.928 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:51:08.928 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:51:08.928 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:51:08.928 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:51:08.928 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:51:08.928 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:51:08.928 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x337e9189", 2024-09-13T15:51:08.928 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:51:08.928 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:51:08.928 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:51:08.928 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:51:08.928 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:51:08.929 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:51:08.929 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.929 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:51:08.929 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.929 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:51:08.929 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.929 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:51:08.929 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:51:08.929 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.929 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.929 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.929 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:51:08.929 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:51:08.929 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.930 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:51:08.930 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:51:08.930 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:51:08.930 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:51:08.930 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:51:08.930 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.930 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.930 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.930 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.930 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:51:08.930 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.930 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:51:08.930 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:51:08.931 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:51:08.931 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:51:08.931 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:51:08.931 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:51:08.931 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:51:08.931 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:51:08.931 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:51:08.931 INFO:tasks.workunit.client.0.smithi112.stdout: "size": "????", 2024-09-13T15:51:08.931 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:51:08.931 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:51:08.931 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:51:08.931 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.931 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.931 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.931 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.932 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.932 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.932 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:51:08.932 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:51:08.932 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:51:08.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2024-09-13T15:51:08.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: '[' primary = replica ']' 2024-09-13T15:51:08.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:811: _scrub_snaps_multi: scruberrors=30 2024-09-13T15:51:08.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:812: _scrub_snaps_multi: jq .inconsistents 2024-09-13T15:51:08.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:812: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-13T15:51:08.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2024-09-13T15:51:08.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-13T15:51:08.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1067: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-13T15:51:08.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-13T15:51:08.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: test no = yes 2024-09-13T15:51:08.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1073: _scrub_snaps_multi: test '' = yes 2024-09-13T15:51:08.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2024-09-13T15:51:08.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: find td/osd-scrub-snaps 2024-09-13T15:51:08.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2024-09-13T15:51:08.935 INFO:tasks.workunit.client.0.smithi112.stderr:td/osd-scrub-snaps/osd.0.pid' 2024-09-13T15:51:08.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids= 2024-09-13T15:51:08.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-13T15:51:08.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2024-09-13T15:51:08.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='506921 ' 2024-09-13T15:51:08.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-13T15:51:08.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2024-09-13T15:51:08.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='506324 ' 2024-09-13T15:51:08.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1085: _scrub_snaps_multi: ERRORS=0 2024-09-13T15:51:08.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1089: _scrub_snaps_multi: '[' primary = primary ']' 2024-09-13T15:51:08.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: seq 1 7 2024-09-13T15:51:08.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-13T15:51:08.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap1 2024-09-13T15:51:09.121 INFO:tasks.workunit.client.0.smithi112.stdout:removed pool test snap snap1 2024-09-13T15:51:09.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-13T15:51:09.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap2 2024-09-13T15:51:09.384 INFO:tasks.workunit.client.0.smithi112.stdout:removed pool test snap snap2 2024-09-13T15:51:09.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-13T15:51:09.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap3 2024-09-13T15:51:09.704 INFO:tasks.workunit.client.0.smithi112.stdout:removed pool test snap snap3 2024-09-13T15:51:09.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-13T15:51:09.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap4 2024-09-13T15:51:10.101 INFO:tasks.workunit.client.0.smithi112.stdout:removed pool test snap snap4 2024-09-13T15:51:10.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-13T15:51:10.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap5 2024-09-13T15:51:10.339 INFO:tasks.workunit.client.0.smithi112.stdout:removed pool test snap snap5 2024-09-13T15:51:10.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-13T15:51:10.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap6 2024-09-13T15:51:10.641 INFO:tasks.workunit.client.0.smithi112.stdout:removed pool test snap snap6 2024-09-13T15:51:10.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2024-09-13T15:51:10.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap7 2024-09-13T15:51:10.950 INFO:tasks.workunit.client.0.smithi112.stdout:removed pool test snap snap7 2024-09-13T15:51:10.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1095: _scrub_snaps_multi: sleep 5 2024-09-13T15:51:15.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1096: _scrub_snaps_multi: local -i loop=0 2024-09-13T15:51:15.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1097: _scrub_snaps_multi: ceph pg dump pgs 2024-09-13T15:51:15.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1097: _scrub_snaps_multi: grep -q snaptrim 2024-09-13T15:51:16.275 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:51:16.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1099: _scrub_snaps_multi: ceph pg dump pgs 2024-09-13T15:51:16.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1099: _scrub_snaps_multi: grep -q snaptrim_error 2024-09-13T15:51:16.597 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:51:16.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1101: _scrub_snaps_multi: break 2024-09-13T15:51:16.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: ceph pg dump pgs 2024-09-13T15:51:16.916 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T15:51:16.916 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent+snaptrim_error 2024-09-13T15:51:10.994767+0000 23'56 37:138 [1,0] 1 [1,0] 1 23'56 2024-09-13T15:51:06.461972+0000 0'0 2024-09-13T15:49:30.766364+0000 7 1 periodic scrub scheduled @ 2024-09-14T19:35:40.443250+0000 33 0 2024-09-13T15:51:16.916 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T15:51:16.916 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T15:51:16.916 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:51:16.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-09-13T15:51:16.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 506921 2024-09-13T15:51:16.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-09-13T15:51:16.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 506324 2024-09-13T15:51:16.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2024-09-13T15:51:16.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:51:16.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:51:16.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:51:16.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:51:16.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:51:22.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:51:22.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: declare -a err_strings 2024-09-13T15:51:22.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2024-09-13T15:51:22.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2024-09-13T15:51:22.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2024-09-13T15:51:22.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2024-09-13T15:51:22.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2024-09-13T15:51:22.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2024-09-13T15:51:22.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2024-09-13T15:51:22.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' 2024-09-13T15:51:22.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:51:22.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:51:22.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:51:22.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:51:22.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:51:22.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:51:22.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:51:22.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:51:22.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:51:22.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:51:22.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:51:22.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:51:22.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:51:22.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:51:22.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:51:22.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:51:22.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: declare -a rep_err_strings 2024-09-13T15:51:22.278 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: eval echo '$primary' 2024-09-13T15:51:22.278 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 1 2024-09-13T15:51:22.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: osd=1 2024-09-13T15:51:22.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-09-13T15:51:22.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1147: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2024-09-13T15:51:22.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1149: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:51:22.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1156: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2024-09-13T15:51:22.282 INFO:tasks.workunit.client.0.smithi112.stdout:TEST PASSED 2024-09-13T15:51:22.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: _scrub_snaps_multi: echo 'TEST PASSED' 2024-09-13T15:51:22.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: _scrub_snaps_multi: return 0 2024-09-13T15:51:22.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1181: TEST_scrub_snaps_primary: err=0 2024-09-13T15:51:22.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1182: TEST_scrub_snaps_primary: CEPH_ARGS='--fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-13T15:51:22.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1183: TEST_scrub_snaps_primary: return 0 2024-09-13T15:51:22.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-09-13T15:51:22.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-13T15:51:22.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:51:22.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-13T15:51:22.284 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:51:22.284 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:51:22.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:51:22.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:51:22.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:51:22.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:51:22.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:51:22.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:51:22.293 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:51:22.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:51:22.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:51:22.294 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:51:22.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:51:22.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:51:22.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:51:22.296 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:51:22.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:51:22.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:51:22.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-13T15:51:22.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:51:22.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:51:22.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:51:22.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.471165 2024-09-13T15:51:22.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:51:22.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:51:22.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-09-13T15:51:22.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-09-13T15:51:22.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2024-09-13T15:51:22.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2024-09-13T15:51:22.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-13T15:51:22.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:51:22.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-13T15:51:22.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:51:22.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:51:22.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:51:22.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:51:22.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:51:22.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:51:22.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:51:22.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:51:22.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:51:22.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:51:22.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:51:22.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:51:22.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:51:22.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:51:22.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:51:22.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:51:22.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:51:22.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:51:22.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-13T15:51:22.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:51:22.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:51:22.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:51:22.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.471165 2024-09-13T15:51:22.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:51:22.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:51:22.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2024-09-13T15:51:22.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:51:22.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:51:22.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:51:22.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.471165 2024-09-13T15:51:22.334 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:51:22.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T15:51:22.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:51:22.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-09-13T15:51:22.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_replica td/osd-scrub-snaps 2024-09-13T15:51:22.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1167: TEST_scrub_snaps_replica: local dir=td/osd-scrub-snaps 2024-09-13T15:51:22.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1168: TEST_scrub_snaps_replica: ORIG_ARGS='--fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-13T15:51:22.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1169: TEST_scrub_snaps_replica: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T15:51:22.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1170: TEST_scrub_snaps_replica: _scrub_snaps_multi td/osd-scrub-snaps replica 2024-09-13T15:51:22.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2024-09-13T15:51:22.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local poolname=test 2024-09-13T15:51:22.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local OBJS=16 2024-09-13T15:51:22.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:735: _scrub_snaps_multi: local OSDS=2 2024-09-13T15:51:22.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: local which=replica 2024-09-13T15:51:22.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: TESTDATA=testdata.471165 2024-09-13T15:51:22.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2024-09-13T15:51:22.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2024-09-13T15:51:22.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:51:22.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:51:22.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:51:22.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2024-09-13T15:51:22.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2024-09-13T15:51:22.864 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:51:22.864 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:51:22.864 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:51:22.865 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:51:22.865 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:51:22.865 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:51:22.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:51:22.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-13T15:51:23.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:51:23.127 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:51:23.127 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:51:23.127 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:51:23.127 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:51:23.128 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:51:23.128 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:51:23.128 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:51:23.129 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:51:23.129 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:51:23.129 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:51:23.129 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:51:23.129 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.471165/ceph-mon.a.asok 2024-09-13T15:51:23.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:51:23.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.471165/ceph-mon.a.asok config get fsid 2024-09-13T15:51:23.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:51:23.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:51:23.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:51:23.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:51:23.247 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:51:23.247 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:51:23.247 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:51:23.247 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:51:23.247 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:51:23.247 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:51:23.247 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:51:23.247 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.471165/ceph-mon.a.asok 2024-09-13T15:51:23.248 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:51:23.248 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.471165/ceph-mon.a.asok config get mon_host 2024-09-13T15:51:23.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2024-09-13T15:51:23.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2024-09-13T15:51:23.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:51:23.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:51:23.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:51:23.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2024-09-13T15:51:23.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:51:24.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:51:24.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:51:24.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:51:24.322 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:51:24.323 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:51:24.323 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:51:24.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:51:24.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:51:24.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:51:24.371 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: expr 2 - 1 2024-09-13T15:51:24.372 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: seq 0 1 2024-09-13T15:51:24.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:51:24.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2024-09-13T15:51:24.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2024-09-13T15:51:24.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:51:24.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:51:24.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:51:24.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-13T15:51:24.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T15:51:24.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:51:24.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:51:24.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:51:24.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-13T15:51:24.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-13T15:51:24.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:51:24.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:51:24.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-13T15:51:24.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:51:24.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:51:24.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:51:24.376 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:51:24.376 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:51:24.376 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:51:24.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:51:24.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:51:24.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:51:24.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:51:24.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:51:24.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-13T15:51:24.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-13T15:51:24.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:51:24.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:51:24.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:51:24.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:51:24.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:51:24.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:51:24.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-13T15:51:24.379 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:51:24.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8415fb40-be5c-4cc7-85d9-a811661447ac 2024-09-13T15:51:24.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8415fb40-be5c-4cc7-85d9-a811661447ac' 2024-09-13T15:51:24.379 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 8415fb40-be5c-4cc7-85d9-a811661447ac 2024-09-13T15:51:24.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:51:24.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB8X+RmRhQ2GBAAgxMdlOpGuVfDzxnITutbVQ== 2024-09-13T15:51:24.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB8X+RmRhQ2GBAAgxMdlOpGuVfDzxnITutbVQ=="}' 2024-09-13T15:51:24.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8415fb40-be5c-4cc7-85d9-a811661447ac -i td/osd-scrub-snaps/0/new.json 2024-09-13T15:51:24.703 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:51:24.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-09-13T15:51:24.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8X+RmRhQ2GBAAgxMdlOpGuVfDzxnITutbVQ== --osd-uuid 8415fb40-be5c-4cc7-85d9-a811661447ac 2024-09-13T15:51:24.782 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:51:24.777+0000 7fda713067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:51:24.787 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:51:24.785+0000 7fda713067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:51:24.789 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:51:24.785+0000 7fda713067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:51:24.789 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:51:24.785+0000 7fda713067c0 -1 bdev(0x55fdd60dfc00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:51:24.789 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:51:24.785+0000 7fda713067c0 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-09-13T15:51:33.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-09-13T15:51:33.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:51:33.133 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:51:33.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:51:33.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:51:33.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:51:33.559 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:51:33.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:51:33.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:51:33.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:51:33.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:51:33.604 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:51:33.601+0000 7feda92497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:51:33.627 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:51:33.625+0000 7feda92497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:51:33.629 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:51:33.625+0000 7feda92497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:51:33.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:51:33.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:51:33.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:51:33.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:51:33.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:51:33.890 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:51:33.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:33.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:51:33.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:33.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:34.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:34.691 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:51:34.685+0000 7feda92497c0 -1 Falling back to public interface 2024-09-13T15:51:35.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:35.277 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:51:35.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:35.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:51:35.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:35.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:35.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:36.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:36.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:36.622 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:51:36.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:51:36.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:36.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:36.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:37.717 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:51:37.713+0000 7feda92497c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:51:37.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:37.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:37.963 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:51:37.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:51:37.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:37.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:38.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:39.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:39.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:39.356 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:51:39.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:51:39.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:39.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:39.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:40.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:40.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:40.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:51:40.848 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:51:40.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:40.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:41.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:42.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:42.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:42.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:51:42.201 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:51:42.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:42.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:42.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:43.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:43.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:43.554 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:51:43.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:51:43.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:43.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:43.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:44.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:44.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:44.945 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:51:44.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:51:44.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:44.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:45.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:46.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:46.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:46.289 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:51:46.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:51:46.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:46.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:46.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:47.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:47.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:47.876 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:51:47.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:51:47.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:47.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:48.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:49.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:49.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:49.216 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:51:49.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:51:49.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:49.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:49.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:50.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:50.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:50.740 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:51:50.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:51:50.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:50.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:51.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:52.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:52.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:52.093 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:51:52.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:51:52.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:52.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:52.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:53.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:53.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:53.529 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:51:53.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:51:53.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:53.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:53.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:54.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:54.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:54.914 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:51:54.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:51:54.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:54.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:55.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:56.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:56.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:56.294 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:51:56.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:51:56.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:56.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:57.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:58.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:58.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:58.329 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:51:58.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:51:58.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:58.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:51:58.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:51:59.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:51:59.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:51:59.884 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:51:59.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:51:59.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:51:59.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:52:00.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:01.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:01.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:01.982 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:52:01.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:52:01.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:01.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:52:03.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:04.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:04.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:04.011 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:52:04.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:52:04.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:04.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:52:04.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:05.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:05.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:05.699 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:52:05.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:52:05.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:05.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:52:06.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:07.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:07.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:07.202 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:52:07.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:52:07.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:07.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:52:07.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:08.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:08.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:08.696 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:52:08.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:52:08.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:08.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:52:09.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:10.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:10.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:10.082 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:52:10.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:52:10.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:10.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:52:10.448 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2326115106,v1:127.0.0.1:6803/2326115106] [v2:127.0.0.1:6804/2326115106,v1:127.0.0.1:6805/2326115106] exists,up 8415fb40-be5c-4cc7-85d9-a811661447ac 2024-09-13T15:52:10.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:52:10.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:52:10.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:52:10.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:52:10.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2024-09-13T15:52:10.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2024-09-13T15:52:10.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:52:10.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T15:52:10.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:52:10.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-13T15:52:10.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T15:52:10.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:52:10.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:52:10.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:52:10.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-13T15:52:10.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-13T15:52:10.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:52:10.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:52:10.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-13T15:52:10.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:52:10.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:52:10.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:52:10.453 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:52:10.453 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:52:10.453 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:52:10.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:52:10.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:52:10.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:52:10.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:52:10.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:52:10.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-13T15:52:10.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-13T15:52:10.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:52:10.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:52:10.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:52:10.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:52:10.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:52:10.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T15:52:10.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-13T15:52:10.455 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:52:10.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a52bde2d-db24-4536-80ff-bec501fff8d9 2024-09-13T15:52:10.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a52bde2d-db24-4536-80ff-bec501fff8d9' 2024-09-13T15:52:10.455 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 a52bde2d-db24-4536-80ff-bec501fff8d9 2024-09-13T15:52:10.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:52:10.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCqX+Rm3n6iHBAA9BoutY/dvvxtSSHRtIO+cQ== 2024-09-13T15:52:10.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCqX+Rm3n6iHBAA9BoutY/dvvxtSSHRtIO+cQ=="}' 2024-09-13T15:52:10.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a52bde2d-db24-4536-80ff-bec501fff8d9 -i td/osd-scrub-snaps/1/new.json 2024-09-13T15:52:11.000 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:52:11.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/1/new.json 2024-09-13T15:52:11.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCqX+Rm3n6iHBAA9BoutY/dvvxtSSHRtIO+cQ== --osd-uuid a52bde2d-db24-4536-80ff-bec501fff8d9 2024-09-13T15:52:11.070 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:52:11.068+0000 7fa32c73a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:52:11.135 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:52:11.136+0000 7fa32c73a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:52:11.137 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:52:11.136+0000 7fa32c73a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:52:11.137 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:52:11.136+0000 7fa32c73a7c0 -1 bdev(0x564e5d246800 td/osd-scrub-snaps/1/block) open stat got: (1) Operation not permitted 2024-09-13T15:52:11.137 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:52:11.136+0000 7fa32c73a7c0 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2024-09-13T15:52:19.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2024-09-13T15:52:19.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:52:19.622 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T15:52:19.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T15:52:19.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:52:20.049 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:52:20.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T15:52:20.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:52:20.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:52:20.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:52:20.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:52:20.094 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:52:20.092+0000 7f9afbbed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:52:20.123 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:52:20.124+0000 7f9afbbed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:52:20.125 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:52:20.124+0000 7f9afbbed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:52:20.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T15:52:20.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:52:20.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:52:20.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:52:20.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:52:20.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:20.442 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:52:20.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:52:20.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:20.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:20.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:21.405 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:52:21.404+0000 7f9afbbed7c0 -1 Falling back to public interface 2024-09-13T15:52:21.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:21.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:21.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:52:21.811 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:52:21.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:21.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:22.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:23.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:23.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:23.166 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:52:23.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:52:23.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:23.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:23.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:24.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:24.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:24.506 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:52:24.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:52:24.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:24.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:24.522 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:52:24.520+0000 7f9afbbed7c0 -1 osd.1 0 log_to_monitors true 2024-09-13T15:52:24.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:25.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:25.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:25.861 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:52:25.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:52:25.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:25.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:26.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:27.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:27.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:27.626 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T15:52:27.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T15:52:27.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:27.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:27.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:28.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:28.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:28.986 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T15:52:28.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T15:52:28.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:28.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:29.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:30.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:30.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:30.342 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T15:52:30.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T15:52:30.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:30.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:30.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:31.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:31.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:31.906 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T15:52:31.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T15:52:31.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:31.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:32.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:33.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:33.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:33.263 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T15:52:33.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T15:52:33.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:33.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:33.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:34.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:34.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:34.816 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T15:52:34.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T15:52:34.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:34.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:35.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:36.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:36.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:36.169 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T15:52:36.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T15:52:36.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:36.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:36.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:37.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:37.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:37.743 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T15:52:37.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T15:52:37.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:37.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:38.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:39.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:39.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:39.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T15:52:39.187 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T15:52:39.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:39.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:39.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:40.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:40.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:40.542 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T15:52:40.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T15:52:40.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:40.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:40.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:41.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:41.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:41.904 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T15:52:41.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T15:52:41.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:41.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:42.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:43.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:43.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:43.259 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T15:52:43.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T15:52:43.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:43.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:43.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:44.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:44.869 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T15:52:44.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:44.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T15:52:44.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:44.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:45.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:46.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:46.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:46.500 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T15:52:46.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T15:52:46.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:46.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:47.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:48.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:48.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:48.134 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T15:52:48.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T15:52:48.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:48.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:48.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:49.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:49.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:49.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T15:52:49.869 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T15:52:49.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:49.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:50.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:51.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:51.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:51.511 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T15:52:51.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T15:52:51.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:51.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:52.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:53.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:53.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:53.023 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T15:52:53.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T15:52:53.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:53.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:53.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:54.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:54.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:54.717 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T15:52:54.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T15:52:54.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:54.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:55.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:56.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:56.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:56.182 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T15:52:56.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T15:52:56.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:56.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:56.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:52:57.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:52:57.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:52:57.527 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T15:52:57.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T15:52:57.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:52:57.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:52:57.862 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3678189799,v1:127.0.0.1:6811/3678189799] [v2:127.0.0.1:6812/3678189799,v1:127.0.0.1:6813/3678189799] exists,up a52bde2d-db24-4536-80ff-bec501fff8d9 2024-09-13T15:52:57.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:52:57.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:52:57.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:52:57.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:748: _scrub_snaps_multi: ceph osd set noscrub 2024-09-13T15:52:58.364 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:52:58.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:749: _scrub_snaps_multi: ceph osd set nodeep-scrub 2024-09-13T15:52:58.924 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:52:58.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: create_pool test 1 1 2024-09-13T15:52:58.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-13T15:52:59.559 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'test' created 2024-09-13T15:52:59.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:53:00.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: wait_for_clean 2024-09-13T15:53:00.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:53:00.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:53:00.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:53:00.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:53:00.587 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:53:00.587 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:53:00.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:53:00.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:53:00.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:53:00.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:53:00.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:53:00.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:53:00.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:53:00.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:53:00.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:53:01.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:53:01.081 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:53:01.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:53:01.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:53:01.081 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:53:01.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-13T15:53:01.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-13T15:53:01.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-13T15:53:01.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:53:01.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:53:01.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672969 2024-09-13T15:53:01.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672969 2024-09-13T15:53:01.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672969' 2024-09-13T15:53:01.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:53:01.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-13T15:53:01.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:53:01.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:53:01.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-13T15:53:01.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:53:01.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-13T15:53:01.492 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836502 2024-09-13T15:53:01.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-13T15:53:01.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:53:01.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836502 2024-09-13T15:53:01.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:53:02.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:53:02.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:53:03.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836502 2024-09-13T15:53:03.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:53:04.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T15:53:04.189 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:53:04.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-13T15:53:04.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:53:04.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672969 2024-09-13T15:53:04.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:53:04.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:53:04.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672969 2024-09-13T15:53:04.540 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:53:04.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672969 2024-09-13T15:53:04.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-13T15:53:04.541 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672969 2024-09-13T15:53:04.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:53:04.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672970 -lt 42949672969 2024-09-13T15:53:04.908 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:53:04.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:53:04.909 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:53:05.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:53:05.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:53:05.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:53:05.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:53:05.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:53:05.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:53:05.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:53:05.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:53:05.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:53:05.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:53:05.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:53:05.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:53:06.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:53:06.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:53:06.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:53:06.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: ceph osd dump 2024-09-13T15:53:06.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2024-09-13T15:53:06.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: awk '{ print $2 }' 2024-09-13T15:53:06.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: poolid=1 2024-09-13T15:53:06.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:756: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.471165 bs=1032 count=1 2024-09-13T15:53:06.525 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T15:53:06.525 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T15:53:06.525 INFO:tasks.workunit.client.0.smithi112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 9.0265e-05 s, 11.4 MB/s 2024-09-13T15:53:06.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 1 16 2024-09-13T15:53:06.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:06.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj1 testdata.471165 2024-09-13T15:53:06.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:06.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj2 testdata.471165 2024-09-13T15:53:06.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:06.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj3 testdata.471165 2024-09-13T15:53:07.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:07.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj4 testdata.471165 2024-09-13T15:53:07.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:07.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj5 testdata.471165 2024-09-13T15:53:07.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:07.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj6 testdata.471165 2024-09-13T15:53:07.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:07.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj7 testdata.471165 2024-09-13T15:53:07.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:07.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj8 testdata.471165 2024-09-13T15:53:08.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:08.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj9 testdata.471165 2024-09-13T15:53:08.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:08.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj10 testdata.471165 2024-09-13T15:53:08.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:08.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj11 testdata.471165 2024-09-13T15:53:09.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:09.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj12 testdata.471165 2024-09-13T15:53:09.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:09.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj13 testdata.471165 2024-09-13T15:53:09.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:09.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj14 testdata.471165 2024-09-13T15:53:09.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:09.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj15 testdata.471165 2024-09-13T15:53:09.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-09-13T15:53:09.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj16 testdata.471165 2024-09-13T15:53:10.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: get_primary test obj1 2024-09-13T15:53:10.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-13T15:53:10.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T15:53:10.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-13T15:53:10.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:53:10.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: local primary=1 2024-09-13T15:53:10.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: get_not_primary test obj1 2024-09-13T15:53:10.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2024-09-13T15:53:10.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-13T15:53:10.408 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2024-09-13T15:53:10.408 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-13T15:53:10.408 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T15:53:10.408 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-13T15:53:10.408 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:53:10.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T15:53:10.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T15:53:10.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2024-09-13T15:53:11.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: local replica=0 2024-09-13T15:53:11.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.471165 '$replica' 2024-09-13T15:53:11.180 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.471165 0 2024-09-13T15:53:11.180 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-09-13T15:53:11.180 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-09-13T15:53:11.180 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.471165 2024-09-13T15:53:11.181 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2024-09-13T15:53:11.181 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-09-13T15:53:11.181 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-09-13T15:53:11.653 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap1 2024-09-13T15:53:11.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=1 2024-09-13T15:53:11.659 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T15:53:11.660 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T15:53:11.660 INFO:tasks.workunit.client.0.smithi112.stderr:256 bytes copied, 0.00027678 s, 925 kB/s 2024-09-13T15:53:11.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.471165 2024-09-13T15:53:12.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.471165 2024-09-13T15:53:12.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.471165 2024-09-13T15:53:12.471 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-09-13T15:53:12.472 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:53:12.472 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.471165 2024-09-13T15:53:12.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:53:12.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.471165 2024-09-13T15:53:12.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:53:12.864 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.471165 2024-09-13T15:53:13.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:53:13.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.471165 2024-09-13T15:53:13.315 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:53:13.315 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.471165 2024-09-13T15:53:13.549 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:53:13.549 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.471165 2024-09-13T15:53:13.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:53:13.809 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.471165 2024-09-13T15:53:14.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:53:14.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.471165 2024-09-13T15:53:14.226 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-09-13T15:53:14.226 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.471165 2024-09-13T15:53:14.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-09-13T15:53:14.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-09-13T15:53:14.661 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap2 2024-09-13T15:53:14.664 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=2 2024-09-13T15:53:14.666 INFO:tasks.workunit.client.0.smithi112.stderr:2+0 records in 2024-09-13T15:53:14.666 INFO:tasks.workunit.client.0.smithi112.stderr:2+0 records out 2024-09-13T15:53:14.666 INFO:tasks.workunit.client.0.smithi112.stderr:512 bytes copied, 0.000206039 s, 2.5 MB/s 2024-09-13T15:53:14.666 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.471165 2024-09-13T15:53:15.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-09-13T15:53:15.046 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-09-13T15:53:15.351 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap3 2024-09-13T15:53:15.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=3 2024-09-13T15:53:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:3+0 records in 2024-09-13T15:53:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:3+0 records out 2024-09-13T15:53:15.357 INFO:tasks.workunit.client.0.smithi112.stderr:768 bytes copied, 0.000275443 s, 2.8 MB/s 2024-09-13T15:53:15.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.471165 2024-09-13T15:53:15.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-09-13T15:53:15.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-09-13T15:53:15.940 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap4 2024-09-13T15:53:15.943 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=4 2024-09-13T15:53:15.944 INFO:tasks.workunit.client.0.smithi112.stderr:4+0 records in 2024-09-13T15:53:15.945 INFO:tasks.workunit.client.0.smithi112.stderr:4+0 records out 2024-09-13T15:53:15.945 INFO:tasks.workunit.client.0.smithi112.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000186468 s, 5.5 MB/s 2024-09-13T15:53:15.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.471165 2024-09-13T15:53:16.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.471165 2024-09-13T15:53:16.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-09-13T15:53:16.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-09-13T15:53:16.784 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap5 2024-09-13T15:53:16.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-09-13T15:53:16.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-09-13T15:53:17.072 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap6 2024-09-13T15:53:17.077 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=6 2024-09-13T15:53:17.079 INFO:tasks.workunit.client.0.smithi112.stderr:6+0 records in 2024-09-13T15:53:17.079 INFO:tasks.workunit.client.0.smithi112.stderr:6+0 records out 2024-09-13T15:53:17.079 INFO:tasks.workunit.client.0.smithi112.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 0.000313555 s, 4.9 MB/s 2024-09-13T15:53:17.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.471165 2024-09-13T15:53:17.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-09-13T15:53:17.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-09-13T15:53:17.721 INFO:tasks.workunit.client.0.smithi112.stdout:created pool test snap snap7 2024-09-13T15:53:17.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-09-13T15:53:18.030 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-09-13T15:53:18.230 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-09-13T15:53:18.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-09-13T15:53:18.424 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:53:18.424 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:53:18.424 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:53:18.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:53:18.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:53:18.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:53:18.736 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2024-09-13T15:53:19.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:53:19.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-09-13T15:53:21.401 INFO:tasks.workunit.client.0.smithi112.stdout:WARNING: only removing head with clones present 2024-09-13T15:53:21.402 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-13T15:53:21.938 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-13T15:53:21.939 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-09-13T15:53:22.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:53:22.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-13T15:53:24.271 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:c52c9666:::obj5:2# 2024-09-13T15:53:24.809 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-13T15:53:24.810 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-09-13T15:53:25.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:53:25.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:53:25.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-13T15:53:26.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-13T15:53:26.814 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:53:26.815 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-13T15:53:26.815 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:26.815 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:53:26.815 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-13T15:53:26.815 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-13T15:53:26.815 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-13T15:53:26.815 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-13T15:53:26.815 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-13T15:53:26.815 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-13T15:53:26.815 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:26.815 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-13T15:53:26.815 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-13T15:53:26.816 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-13T15:53:26.816 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-13T15:53:26.816 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:53:26.816 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:53:26.816 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:26.816 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:26.816 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:53:26.816 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:53:26.816 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:26.816 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:53:26.816 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:26.816 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:53:26.817 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:26.817 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:53:26.817 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:26.817 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:53:26.817 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:26.817 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:53:26.817 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:53:26.817 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:53:26.817 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:26.817 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:53:26.817 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:53:26.817 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:53:26.817 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:26.817 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:53:26.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-13T15:53:26.818 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:53:26.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-13T15:53:28.374 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:c52c9666:::obj5:1# 2024-09-13T15:53:28.908 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-13T15:53:30.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-13T15:53:30.014 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:53:30.014 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-13T15:53:30.015 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:30.015 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:53:30.015 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-13T15:53:30.015 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-13T15:53:30.015 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-13T15:53:30.015 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-13T15:53:30.015 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-13T15:53:30.016 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-13T15:53:30.016 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:30.016 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-13T15:53:30.016 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-13T15:53:30.016 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-13T15:53:30.016 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-13T15:53:30.017 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:53:30.017 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:53:30.017 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:30.017 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:30.017 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:53:30.017 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:53:30.018 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:30.018 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:53:30.018 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:30.018 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:53:30.018 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:30.019 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:53:30.019 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:30.019 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:53:30.019 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:30.019 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:53:30.019 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:53:30.020 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:53:30.020 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:30.020 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:53:30.020 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:53:30.020 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-09-13T15:53:30.020 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:30.021 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:53:30.021 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-13T15:53:30.021 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:53:30.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-13T15:53:30.022 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-09-13T15:53:30.023 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-09-13T15:53:30.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:53:30.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=18 2024-09-13T15:53:30.915 INFO:tasks.workunit.client.0.smithi112.stderr:18+0 records in 2024-09-13T15:53:30.915 INFO:tasks.workunit.client.0.smithi112.stderr:18+0 records out 2024-09-13T15:53:30.915 INFO:tasks.workunit.client.0.smithi112.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000247533 s, 18.6 MB/s 2024-09-13T15:53:30.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.471165 2024-09-13T15:53:32.812 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2024-09-13T15:53:33.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:53:33.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=15 2024-09-13T15:53:33.726 INFO:tasks.workunit.client.0.smithi112.stderr:15+0 records in 2024-09-13T15:53:33.726 INFO:tasks.workunit.client.0.smithi112.stderr:15+0 records out 2024-09-13T15:53:33.726 INFO:tasks.workunit.client.0.smithi112.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000211388 s, 18.2 MB/s 2024-09-13T15:53:33.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.471165 2024-09-13T15:53:35.634 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2024-09-13T15:53:35.635 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-09-13T15:53:36.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:53:36.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-13T15:53:37.976 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:0ee9ae15:::obj4:7# 2024-09-13T15:53:39.016 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-13T15:53:40.137 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-13T15:53:40.139 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-13T15:53:40.139 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:40.139 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:53:40.139 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-13T15:53:40.139 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-13T15:53:40.139 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-13T15:53:40.139 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-13T15:53:40.139 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-13T15:53:40.139 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-13T15:53:40.139 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:40.139 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-13T15:53:40.140 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-13T15:53:40.140 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-13T15:53:40.140 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-13T15:53:40.140 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:53:40.140 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:40.140 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:40.140 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:53:40.140 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:40.140 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:53:40.140 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:40.140 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:40.140 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:53:40.140 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:40.141 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:40.141 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:53:40.141 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:53:40.141 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:40.141 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:53:40.141 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:53:40.141 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:40.141 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:53:40.141 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-13T15:53:40.141 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:40.142 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:40.142 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:40.142 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:40.142 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:40.142 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:40.142 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-09-13T15:53:40.142 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2024-09-13T15:53:40.142 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-09-13T15:53:41.053 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:53:41.053 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-09-13T15:53:42.387 INFO:tasks.workunit.client.0.smithi112.stdout:remove #1:461f8b5e:::obj16:7# 2024-09-13T15:53:42.924 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-09-13T15:53:44.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-09-13T15:53:44.029 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-09-13T15:53:44.029 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-09-13T15:53:44.029 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-09-13T15:53:44.029 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-09-13T15:53:44.029 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-09-13T15:53:44.029 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-09-13T15:53:44.029 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-09-13T15:53:44.030 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-09-13T15:53:44.030 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:44.030 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-09-13T15:53:44.030 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-09-13T15:53:44.030 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-09-13T15:53:44.030 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-09-13T15:53:44.030 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:53:44.030 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:44.030 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-09-13T15:53:44.030 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:53:44.030 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:44.030 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-09-13T15:53:44.030 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-09-13T15:53:44.031 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:53:44.031 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:53:44.031 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-09-13T15:53:44.031 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:53:44.031 INFO:tasks.workunit.client.0.smithi112.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-09-13T15:53:44.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-09-13T15:53:44.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-09-13T15:53:44.032 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2024-09-13T15:53:44.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:53:44.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-09-13T15:53:46.734 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:53:46.735 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-09-13T15:53:46.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:53:46.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.471165 bs=256 count=7 2024-09-13T15:53:46.738 INFO:tasks.workunit.client.0.smithi112.stderr:7+0 records in 2024-09-13T15:53:46.738 INFO:tasks.workunit.client.0.smithi112.stderr:7+0 records out 2024-09-13T15:53:46.738 INFO:tasks.workunit.client.0.smithi112.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 0.000304309 s, 5.9 MB/s 2024-09-13T15:53:46.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.471165 2024-09-13T15:53:48.653 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2024-09-13T15:53:49.028 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:53:49.569 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:53:49.569 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-09-13T15:53:51.414 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2024-09-13T15:53:51.822 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:53:52.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:53:52.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-09-13T15:53:54.493 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2024-09-13T15:53:54.853 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:53:55.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:53:55.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-09-13T15:53:57.149 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2024-09-13T15:53:57.610 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:53:57.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:53:57.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-09-13T15:53:59.620 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2024-09-13T15:53:59.970 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:54:00.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:54:00.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-09-13T15:54:02.886 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2024-09-13T15:54:03.342 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:54:03.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:54:03.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-09-13T15:54:05.360 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2024-09-13T15:54:05.722 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:54:06.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:54:06.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-09-13T15:54:07.770 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2024-09-13T15:54:08.136 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:54:08.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:54:08.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-09-13T15:54:10.477 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2024-09-13T15:54:10.846 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:54:11.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:54:11.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-09-13T15:54:13.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-09-13T15:54:13.354 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2024-09-13T15:54:13.727 INFO:tasks.workunit.client.0.smithi112.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-09-13T15:54:14.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T15:54:14.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-09-13T15:54:15.835 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-09-13T15:54:15.837 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-09-13T15:54:15.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: rm -f testdata.471165 2024-09-13T15:54:15.840 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: expr 2 - 1 2024-09-13T15:54:15.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: seq 0 1 2024-09-13T15:54:15.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:54:15.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2024-09-13T15:54:15.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2024-09-13T15:54:15.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:54:15.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-13T15:54:15.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:54:15.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-09-13T15:54:15.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T15:54:15.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:54:15.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:54:15.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:54:15.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-09-13T15:54:15.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-09-13T15:54:15.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:54:15.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:54:15.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-13T15:54:15.845 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:54:15.845 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:54:15.845 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:54:15.845 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:54:15.845 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:54:15.845 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:54:15.847 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:54:15.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:54:15.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:54:15.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-13T15:54:15.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-13T15:54:15.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:54:15.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:54:15.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:54:15.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:54:15.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:54:15.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:54:15.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-09-13T15:54:15.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-13T15:54:15.848 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:54:15.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:54:15.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-09-13T15:54:15.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-13T15:54:15.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:54:15.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:54:15.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:54:15.893 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:15.889+0000 7fc56c5d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:54:15.895 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:15.893+0000 7fc56c5d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:54:15.896 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:15.893+0000 7fc56c5d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:54:16.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-13T15:54:16.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:54:16.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:54:16.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:54:16.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:54:16.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:54:16.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:54:16.193 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:54:16.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:54:16.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:54:16.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:54:16.966 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:16.961+0000 7fc56c5d57c0 -1 Falling back to public interface 2024-09-13T15:54:17.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:54:17.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:54:17.582 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:54:17.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:54:17.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:54:17.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:54:17.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:54:18.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:54:18.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:54:18.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:54:18.926 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:54:18.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:54:18.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:54:19.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:54:19.868 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:19.864+0000 7fc56c5d57c0 -1 osd.0 23 log_to_monitors true 2024-09-13T15:54:20.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:54:20.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:54:20.289 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:54:20.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:54:20.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:54:20.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:54:20.651 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 26 up_thru 0 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/2756472039,v1:127.0.0.1:6803/2756472039] [v2:127.0.0.1:6804/2756472039,v1:127.0.0.1:6805/2756472039] exists,up 8415fb40-be5c-4cc7-85d9-a811661447ac 2024-09-13T15:54:20.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:54:20.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:54:20.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:54:20.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:54:20.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2024-09-13T15:54:20.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2024-09-13T15:54:20.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T15:54:20.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T15:54:20.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T15:54:20.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/1 2024-09-13T15:54:20.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-09-13T15:54:20.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:54:20.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:54:20.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:54:20.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-09-13T15:54:20.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-09-13T15:54:20.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T15:54:20.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T15:54:20.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-09-13T15:54:20.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T15:54:20.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:54:20.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:54:20.659 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:54:20.659 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:54:20.659 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:54:20.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:54:20.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' 2024-09-13T15:54:20.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:54:20.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-09-13T15:54:20.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-09-13T15:54:20.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:54:20.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:54:20.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:54:20.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:54:20.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T15:54:20.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T15:54:20.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/1 2024-09-13T15:54:20.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T15:54:20.660 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:54:20.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.471165/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T15:54:20.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/1/whoami 2024-09-13T15:54:20.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T15:54:20.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T15:54:20.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T15:54:20.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T15:54:20.703 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:20.700+0000 7fbdcd9337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:54:20.771 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:20.768+0000 7fbdcd9337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:54:20.773 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:20.768+0000 7fbdcd9337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:54:21.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T15:54:21.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:54:21.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:54:21.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:54:21.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:54:21.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:54:21.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:54:21.038 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:54:21.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:54:21.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:54:21.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:54:22.286 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:22.280+0000 7fbdcd9337c0 -1 Falling back to public interface 2024-09-13T15:54:22.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:54:22.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:54:22.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:54:22.470 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:54:22.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:54:22.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:54:22.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:54:23.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:54:23.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:54:23.827 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:54:23.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:54:23.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:54:23.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:54:24.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:54:25.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:54:25.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:54:25.180 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:54:25.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:54:25.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:54:25.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:54:25.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:54:26.153 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:26.148+0000 7fbdcd9337c0 -1 osd.1 23 log_to_monitors true 2024-09-13T15:54:26.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:54:26.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:54:26.639 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:54:26.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:54:26.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:54:26.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:54:27.149 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 29 up_thru 14 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/3259212112,v1:127.0.0.1:6811/3259212112] [v2:127.0.0.1:6812/3259212112,v1:127.0.0.1:6813/3259212112] exists,up a52bde2d-db24-4536-80ff-bec501fff8d9 2024-09-13T15:54:27.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:54:27.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:54:27.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:54:27.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2024-09-13T15:54:27.303 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T15:54:27.303 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-13T15:54:27.303 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:54:27.431 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T15:54:27.431 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-13T15:54:27.432 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:54:27.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2024-09-13T15:54:27.656 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T15:54:27.656 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-13T15:54:27.656 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:54:27.674 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T15:54:27.674 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-13T15:54:27.675 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:54:27.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2024-09-13T15:54:27.869 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T15:54:27.869 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-13T15:54:27.869 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:54:27.883 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T15:54:27.883 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-09-13T15:54:27.883 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:54:27.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:777: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2024-09-13T15:54:28.052 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T15:54:28.052 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-09-13T15:54:28.052 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:54:28.065 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T15:54:28.065 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' (not observed, change may require restart) " 2024-09-13T15:54:28.065 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:54:28.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2024-09-13T15:54:28.237 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T15:54:28.237 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-09-13T15:54:28.237 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:54:28.250 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T15:54:28.250 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' (not observed, change may require restart) " 2024-09-13T15:54:28.250 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:54:28.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: wait_for_clean 2024-09-13T15:54:28.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:54:28.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:54:28.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:54:28.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:54:28.267 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:54:28.267 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:54:28.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:54:28.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:54:28.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:54:28.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:54:28.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:54:28.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:54:28.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:54:28.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:54:28.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:54:28.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:54:28.697 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T15:54:28.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:54:28.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:54:28.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:54:28.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149699 2024-09-13T15:54:28.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149699 2024-09-13T15:54:28.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149699' 2024-09-13T15:54:28.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:54:28.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:54:29.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051586 2024-09-13T15:54:29.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051586 2024-09-13T15:54:29.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149699 1-124554051586' 2024-09-13T15:54:29.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:54:29.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149699 2024-09-13T15:54:29.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:54:29.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:54:29.089 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149699 2024-09-13T15:54:29.089 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:54:29.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149699 2024-09-13T15:54:29.090 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 111669149699 2024-09-13T15:54:29.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149699' 2024-09-13T15:54:29.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:54:29.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149699 -lt 111669149699 2024-09-13T15:54:29.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:54:29.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-124554051586 2024-09-13T15:54:29.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:54:29.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:54:29.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-124554051586 2024-09-13T15:54:29.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:54:29.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051586 2024-09-13T15:54:29.445 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 124554051586 2024-09-13T15:54:29.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-09-13T15:54:29.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:54:29.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-13T15:54:29.828 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:54:29.829 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:54:29.829 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:54:30.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:54:30.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:54:30.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:54:30.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:54:30.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:54:30.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:54:30.263 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:54:30.263 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:54:30.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:54:30.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:54:30.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:54:30.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:54:31.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:54:31.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:54:31.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:54:31.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:781: _scrub_snaps_multi: local pgid=1.0 2024-09-13T15:54:31.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:782: _scrub_snaps_multi: pg_scrub 1.0 2024-09-13T15:54:31.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-13T15:54:31.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-13T15:54:31.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:54:31.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:54:31.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:54:31.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:54:31.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-13T15:52:59.451385+0000 2024-09-13T15:54:31.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-13T15:54:31.756 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.0 on osd.1 to scrub 2024-09-13T15:54:31.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-13T15:52:59.451385+0000 2024-09-13T15:54:31.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T15:54:31.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T15:52:59.451385+0000 2024-09-13T15:54:31.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T15:54:31.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T15:54:31.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:54:31.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:54:31.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:54:31.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:54:31.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:54:31.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:54:32.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:52:59.451385+0000 '>' 2024-09-13T15:52:59.451385+0000 2024-09-13T15:54:32.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:54:33.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:54:33.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:54:33.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:54:33.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:54:33.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:54:33.119 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:54:33.119 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:54:33.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:52:59.451385+0000 '>' 2024-09-13T15:52:59.451385+0000 2024-09-13T15:54:33.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:54:34.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:54:34.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:54:34.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:54:34.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:54:34.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:54:34.492 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:54:34.492 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:54:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:52:59.451385+0000 '>' 2024-09-13T15:52:59.451385+0000 2024-09-13T15:54:34.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:54:35.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:54:35.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:54:35.829 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:54:35.829 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:54:35.830 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:54:35.830 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:54:35.830 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:54:36.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:52:59.451385+0000 '>' 2024-09-13T15:52:59.451385+0000 2024-09-13T15:54:36.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T15:54:37.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T15:54:37.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-13T15:54:37.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T15:54:37.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T15:54:37.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T15:54:37.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T15:54:37.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T15:54:37.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T15:54:33.181331+0000 '>' 2024-09-13T15:52:59.451385+0000 2024-09-13T15:54:37.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T15:54:37.694 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:54:37.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: wc -l 2024-09-13T15:54:37.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: test 16 -gt 3 2024-09-13T15:54:37.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:54:37.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: wc -l 2024-09-13T15:54:37.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: test 16 -gt 3 2024-09-13T15:54:37.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:789: _scrub_snaps_multi: rados list-inconsistent-pg test 2024-09-13T15:54:37.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2024-09-13T15:54:37.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: test 1 = 1 2024-09-13T15:54:37.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2024-09-13T15:54:37.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: test 1.0 = 1.0 2024-09-13T15:54:37.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2024-09-13T15:54:37.697 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:54:37.698 INFO:tasks.workunit.client.0.smithi112.stdout: "epoch": 29, 2024-09-13T15:54:37.698 INFO:tasks.workunit.client.0.smithi112.stdout: "inconsistents": [ 2024-09-13T15:54:37.698 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.698 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.698 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj5", 2024-09-13T15:54:37.698 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.698 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.698 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:54:37.698 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 5 2024-09-13T15:54:37.698 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.698 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.698 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:54:37.698 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:54:37.699 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.699 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.699 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.699 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj5", 2024-09-13T15:54:37.699 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.699 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": 1, 2024-09-13T15:54:37.699 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1718170787, 2024-09-13T15:54:37.699 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.699 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.699 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.699 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.699 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'19", 2024-09-13T15:54:37.699 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'5", 2024-09-13T15:54:37.700 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4269.0:1", 2024-09-13T15:54:37.700 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 5, 2024-09-13T15:54:37.700 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.700 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:07.402708+0000", 2024-09-13T15:54:37.700 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:07.403423+0000", 2024-09-13T15:54:37.700 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.700 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.700 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.700 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.700 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.700 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.700 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.701 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0xfacba517", 2024-09-13T15:54:37.701 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.701 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.701 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.701 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.701 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.701 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.701 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.701 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.701 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.701 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.701 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.701 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.702 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.702 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.702 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:54:37.702 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.702 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.702 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.702 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.702 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.702 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.702 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032 2024-09-13T15:54:37.702 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.703 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.703 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.703 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.703 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.703 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj5", 2024-09-13T15:54:37.703 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.703 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.703 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 2, 2024-09-13T15:54:37.703 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 20 2024-09-13T15:54:37.703 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.703 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.703 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:54:37.703 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:54:37.704 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.704 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.704 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.704 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj5", 2024-09-13T15:54:37.704 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.704 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": 2, 2024-09-13T15:54:37.704 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1718170787, 2024-09-13T15:54:37.704 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.704 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.704 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.704 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.704 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "18'41", 2024-09-13T15:54:37.704 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "17'20", 2024-09-13T15:54:37.704 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4317.0:1", 2024-09-13T15:54:37.705 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 20, 2024-09-13T15:54:37.705 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.705 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:12.091728+0000", 2024-09-13T15:54:37.705 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:12.092577+0000", 2024-09-13T15:54:37.705 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.705 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.705 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.705 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.705 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.705 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.705 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.705 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x4924c9ab", 2024-09-13T15:54:37.705 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.705 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.706 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.706 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.706 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.706 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.706 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.706 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.706 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.706 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.706 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.706 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.707 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.707 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.707 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:54:37.707 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.707 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.707 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.707 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.707 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.707 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.707 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256 2024-09-13T15:54:37.707 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.707 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.707 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.708 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.708 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.708 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj5", 2024-09-13T15:54:37.708 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.708 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.708 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 4, 2024-09-13T15:54:37.708 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 42 2024-09-13T15:54:37.708 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.708 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.708 INFO:tasks.workunit.client.0.smithi112.stdout: "size_mismatch" 2024-09-13T15:54:37.708 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.708 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:54:37.708 INFO:tasks.workunit.client.0.smithi112.stdout: "size_mismatch_info", 2024-09-13T15:54:37.709 INFO:tasks.workunit.client.0.smithi112.stdout: "obj_size_info_mismatch" 2024-09-13T15:54:37.709 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.709 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.709 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.709 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj5", 2024-09-13T15:54:37.709 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.709 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": 4, 2024-09-13T15:54:37.709 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1718170787, 2024-09-13T15:54:37.709 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.709 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.709 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.709 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.710 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "20'45", 2024-09-13T15:54:37.710 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "18'42", 2024-09-13T15:54:37.710 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4353.0:1", 2024-09-13T15:54:37.710 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 42, 2024-09-13T15:54:37.710 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 512, 2024-09-13T15:54:37.710 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:14.712509+0000", 2024-09-13T15:54:37.710 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:14.759104+0000", 2024-09-13T15:54:37.710 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.711 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.711 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.711 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.711 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.711 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.711 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.711 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x9547a317", 2024-09-13T15:54:37.711 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.711 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.711 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.711 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.711 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.711 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.711 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.712 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.712 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.712 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.712 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.712 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.712 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.712 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.712 INFO:tasks.workunit.client.0.smithi112.stdout: "size_mismatch_info", 2024-09-13T15:54:37.712 INFO:tasks.workunit.client.0.smithi112.stdout: "obj_size_info_mismatch" 2024-09-13T15:54:37.712 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.712 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 4608, 2024-09-13T15:54:37.712 INFO:tasks.workunit.client.0.smithi112.stdout: "object_info": { 2024-09-13T15:54:37.712 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.713 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj5", 2024-09-13T15:54:37.713 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.713 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": 4, 2024-09-13T15:54:37.713 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1718170787, 2024-09-13T15:54:37.713 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.713 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.713 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.713 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.713 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "20'45", 2024-09-13T15:54:37.713 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "18'42", 2024-09-13T15:54:37.713 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4353.0:1", 2024-09-13T15:54:37.713 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 42, 2024-09-13T15:54:37.713 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 512, 2024-09-13T15:54:37.714 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:14.712509+0000", 2024-09-13T15:54:37.714 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:14.759104+0000", 2024-09-13T15:54:37.714 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.714 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.714 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.714 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.714 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.714 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.714 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.714 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x9547a317", 2024-09-13T15:54:37.714 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.714 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.714 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.714 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.715 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.715 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.715 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.715 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.715 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.715 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.715 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.715 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.715 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.715 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.715 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 512 2024-09-13T15:54:37.715 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.716 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.716 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.716 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.716 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.716 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj4", 2024-09-13T15:54:37.716 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.716 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.716 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 7, 2024-09-13T15:54:37.716 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 4 2024-09-13T15:54:37.717 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.717 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.717 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:54:37.717 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:54:37.717 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.717 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.717 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.717 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj4", 2024-09-13T15:54:37.717 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.717 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": 7, 2024-09-13T15:54:37.717 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 2826278768, 2024-09-13T15:54:37.717 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.717 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.717 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.718 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.718 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "23'51", 2024-09-13T15:54:37.718 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'4", 2024-09-13T15:54:37.718 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4266.0:1", 2024-09-13T15:54:37.718 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 4, 2024-09-13T15:54:37.718 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.718 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:07.224022+0000", 2024-09-13T15:54:37.718 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:07.224779+0000", 2024-09-13T15:54:37.718 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.718 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.718 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.718 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.718 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.719 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.719 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.719 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0xfacba517", 2024-09-13T15:54:37.719 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.719 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.719 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.719 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.719 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.719 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.719 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.719 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.719 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.719 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.719 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.720 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.720 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.720 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.720 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:54:37.720 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.720 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.720 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.720 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.720 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.720 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.720 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032 2024-09-13T15:54:37.720 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.720 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.720 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.721 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.721 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.721 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj5", 2024-09-13T15:54:37.721 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.722 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.722 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 7, 2024-09-13T15:54:37.722 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 0 2024-09-13T15:54:37.722 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.722 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.723 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:54:37.723 INFO:tasks.workunit.client.0.smithi112.stdout: "missing", 2024-09-13T15:54:37.723 INFO:tasks.workunit.client.0.smithi112.stdout: "info_missing" 2024-09-13T15:54:37.723 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.723 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.723 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.723 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.723 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.723 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.724 INFO:tasks.workunit.client.0.smithi112.stdout: "info_missing" 2024-09-13T15:54:37.724 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.724 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1792 2024-09-13T15:54:37.724 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.724 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.724 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.724 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.724 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.724 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:54:37.725 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.725 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.725 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.725 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.725 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.725 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.726 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj1", 2024-09-13T15:54:37.726 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.726 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.726 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:54:37.726 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 18 2024-09-13T15:54:37.726 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.726 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.726 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:54:37.726 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:54:37.726 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.727 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.727 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.727 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj1", 2024-09-13T15:54:37.727 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.727 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:54:37.727 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1828249343, 2024-09-13T15:54:37.727 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.727 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.727 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.727 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.727 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'18", 2024-09-13T15:54:37.728 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'1", 2024-09-13T15:54:37.728 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4314.0:1", 2024-09-13T15:54:37.728 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 18, 2024-09-13T15:54:37.728 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.728 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:11.705609+0000", 2024-09-13T15:54:37.728 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:11.831285+0000", 2024-09-13T15:54:37.728 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.728 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.728 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.728 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.728 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.729 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.729 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.729 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x4924c9ab", 2024-09-13T15:54:37.729 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.729 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.729 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.729 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.729 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.729 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.730 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.730 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.730 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.730 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.730 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.730 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.730 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.731 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.731 INFO:tasks.workunit.client.0.smithi112.stdout: "missing" 2024-09-13T15:54:37.731 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.731 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.731 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.731 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.731 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.732 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.732 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256 2024-09-13T15:54:37.732 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.732 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.732 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.732 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.733 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.733 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj10", 2024-09-13T15:54:37.733 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.733 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.733 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:54:37.733 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 32 2024-09-13T15:54:37.734 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.734 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.734 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:54:37.735 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.735 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:54:37.735 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.735 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.735 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj10", 2024-09-13T15:54:37.735 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.735 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:54:37.735 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 718195851, 2024-09-13T15:54:37.735 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.735 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.735 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.735 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.735 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'32", 2024-09-13T15:54:37.735 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'10", 2024-09-13T15:54:37.736 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4335.0:1", 2024-09-13T15:54:37.736 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 32, 2024-09-13T15:54:37.736 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.736 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:13.360715+0000", 2024-09-13T15:54:37.736 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:13.361428+0000", 2024-09-13T15:54:37.736 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.736 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.736 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.736 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.736 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.736 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.736 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.736 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x4924c9ab", 2024-09-13T15:54:37.736 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.737 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.737 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.737 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.737 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.737 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.737 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.737 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.737 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.737 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.737 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.737 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.737 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.737 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.737 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.738 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.738 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:54:37.738 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:54:37.738 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.738 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:54:37.738 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.738 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "????", 2024-09-13T15:54:37.738 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:54:37.738 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:54:37.738 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.738 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.738 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.738 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.739 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.739 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.739 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.739 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.739 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.739 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.739 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.739 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:54:37.739 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:54:37.739 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.739 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:54:37.739 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.739 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:54:37.739 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:54:37.739 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:54:37.740 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.740 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.740 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.740 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.740 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.740 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.740 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.740 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.740 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.740 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj11", 2024-09-13T15:54:37.740 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.740 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.740 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:54:37.740 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 34 2024-09-13T15:54:37.741 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.741 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.741 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:54:37.741 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.741 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:54:37.741 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.741 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.741 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj11", 2024-09-13T15:54:37.741 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.741 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:54:37.741 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 693400951, 2024-09-13T15:54:37.741 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.741 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.741 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.742 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.742 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'34", 2024-09-13T15:54:37.742 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'11", 2024-09-13T15:54:37.742 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4338.0:1", 2024-09-13T15:54:37.742 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 34, 2024-09-13T15:54:37.742 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.742 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:13.612963+0000", 2024-09-13T15:54:37.742 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:13.613702+0000", 2024-09-13T15:54:37.742 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.742 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.742 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.742 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.742 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.742 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.743 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.743 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x4924c9ab", 2024-09-13T15:54:37.743 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.743 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.743 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.743 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.743 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.743 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.743 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.743 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.743 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.743 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.743 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.743 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.744 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.744 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.744 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.744 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.744 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:54:37.744 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [] 2024-09-13T15:54:37.744 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.744 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.744 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.744 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.744 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.744 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.744 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.744 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.745 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:54:37.745 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:54:37.745 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.745 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:54:37.745 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.745 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:54:37.745 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:54:37.745 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:54:37.745 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.745 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.745 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.745 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.745 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.745 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.746 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.746 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.746 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.746 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj13", 2024-09-13T15:54:37.746 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.746 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.746 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:54:37.746 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 38 2024-09-13T15:54:37.746 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.746 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.746 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:54:37.746 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.746 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:54:37.746 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.747 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.747 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj13", 2024-09-13T15:54:37.747 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.747 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:54:37.747 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 2087409765, 2024-09-13T15:54:37.747 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.747 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.747 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.747 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.747 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'38", 2024-09-13T15:54:37.747 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'13", 2024-09-13T15:54:37.748 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4344.0:1", 2024-09-13T15:54:37.748 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 38, 2024-09-13T15:54:37.748 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.748 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:14.060871+0000", 2024-09-13T15:54:37.748 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:14.061545+0000", 2024-09-13T15:54:37.748 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.748 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.748 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.748 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.748 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.748 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.749 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.749 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x4924c9ab", 2024-09-13T15:54:37.749 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.749 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.749 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.749 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.749 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.749 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.749 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.749 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.749 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.749 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.749 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.749 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.750 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.750 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.750 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.750 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.750 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:54:37.750 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:54:37.750 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.750 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:54:37.750 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.750 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]" 2024-09-13T15:54:37.750 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.750 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.750 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.750 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.751 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.751 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.751 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.751 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.751 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.751 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.751 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:54:37.751 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:54:37.751 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.751 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:54:37.751 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.751 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:54:37.751 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:54:37.751 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:54:37.752 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.752 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.752 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.752 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.752 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.752 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.752 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.752 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.752 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.752 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj14", 2024-09-13T15:54:37.752 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.752 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.752 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:54:37.753 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 40 2024-09-13T15:54:37.753 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.753 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.753 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:54:37.753 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.753 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:54:37.753 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.753 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.753 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj14", 2024-09-13T15:54:37.753 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.753 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:54:37.753 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 2484217095, 2024-09-13T15:54:37.753 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.753 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.754 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.754 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.754 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'40", 2024-09-13T15:54:37.754 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'14", 2024-09-13T15:54:37.754 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4347.0:1", 2024-09-13T15:54:37.754 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 40, 2024-09-13T15:54:37.754 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.754 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:14.269055+0000", 2024-09-13T15:54:37.754 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:14.269757+0000", 2024-09-13T15:54:37.754 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.754 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.754 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.754 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.754 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.755 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.755 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.755 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x4924c9ab", 2024-09-13T15:54:37.755 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.755 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.755 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.755 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.755 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.755 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.755 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.755 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.755 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.755 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.755 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.756 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.756 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.756 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.756 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.756 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.756 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:54:37.756 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:54:37.756 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.756 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:54:37.756 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1033, 2024-09-13T15:54:37.756 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:54:37.756 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:54:37.756 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:54:37.757 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.757 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.757 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.757 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.757 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.757 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.757 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.757 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.757 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.757 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.757 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.757 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:54:37.758 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:54:37.758 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.758 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:54:37.758 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.758 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:54:37.758 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:54:37.758 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:54:37.758 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.758 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.758 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.758 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.758 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.758 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.758 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.759 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.759 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.759 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj15", 2024-09-13T15:54:37.759 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.759 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.759 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:54:37.759 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 15 2024-09-13T15:54:37.759 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.759 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.759 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:54:37.759 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_corrupted" 2024-09-13T15:54:37.759 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.759 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.760 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.760 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj15", 2024-09-13T15:54:37.760 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.760 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:54:37.760 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 612772309, 2024-09-13T15:54:37.760 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.760 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.760 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.760 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.760 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "16'15", 2024-09-13T15:54:37.760 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "0'0", 2024-09-13T15:54:37.760 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4299.0:1", 2024-09-13T15:54:37.760 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 15, 2024-09-13T15:54:37.761 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.761 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:09.697617+0000", 2024-09-13T15:54:37.761 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:09.698344+0000", 2024-09-13T15:54:37.761 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.761 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.761 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.761 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.761 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.761 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.761 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.761 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0xfacba517", 2024-09-13T15:54:37.761 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.761 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.761 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.761 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.762 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.762 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.762 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.762 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.762 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.762 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.762 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.762 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.762 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.762 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.762 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_corrupted" 2024-09-13T15:54:37.762 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.762 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.762 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": "Z2FyYmFnZQo=" 2024-09-13T15:54:37.763 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.763 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.763 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.763 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.763 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.763 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.763 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.763 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 0, 2024-09-13T15:54:37.763 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [] 2024-09-13T15:54:37.763 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.763 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.763 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.763 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.764 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.764 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.764 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj2", 2024-09-13T15:54:37.764 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.764 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.764 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:54:37.764 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 56 2024-09-13T15:54:37.764 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.764 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.764 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:54:37.764 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_missing" 2024-09-13T15:54:37.764 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.764 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.765 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.765 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj2", 2024-09-13T15:54:37.765 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.765 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:54:37.765 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1058988552, 2024-09-13T15:54:37.765 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.765 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.765 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.765 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.765 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "23'56", 2024-09-13T15:54:37.765 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "20'48", 2024-09-13T15:54:37.765 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4389.0:1", 2024-09-13T15:54:37.765 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 56, 2024-09-13T15:54:37.765 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 0, 2024-09-13T15:54:37.766 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:18.275394+0000", 2024-09-13T15:54:37.766 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:18.276111+0000", 2024-09-13T15:54:37.766 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.766 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.766 INFO:tasks.workunit.client.0.smithi112.stdout: "whiteout", 2024-09-13T15:54:37.766 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty" 2024-09-13T15:54:37.766 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.766 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.766 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.766 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0xffffffff", 2024-09-13T15:54:37.766 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.766 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.766 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.766 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.767 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.767 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.767 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.767 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.767 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.767 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.767 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.767 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.767 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.767 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.767 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_missing" 2024-09-13T15:54:37.767 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.767 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 0 2024-09-13T15:54:37.767 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.768 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.768 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.768 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.768 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.768 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 0, 2024-09-13T15:54:37.768 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.768 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 7, 2024-09-13T15:54:37.768 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:54:37.768 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.768 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 4, 2024-09-13T15:54:37.768 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.768 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:54:37.768 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:54:37.768 INFO:tasks.workunit.client.0.smithi112.stdout: 4, 2024-09-13T15:54:37.769 INFO:tasks.workunit.client.0.smithi112.stdout: 3, 2024-09-13T15:54:37.769 INFO:tasks.workunit.client.0.smithi112.stdout: 2, 2024-09-13T15:54:37.769 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:54:37.769 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.769 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.769 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.769 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 7, 2024-09-13T15:54:37.769 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1024, 2024-09-13T15:54:37.769 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:54:37.769 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:54:37.769 INFO:tasks.workunit.client.0.smithi112.stdout: 7, 2024-09-13T15:54:37.769 INFO:tasks.workunit.client.0.smithi112.stdout: 6, 2024-09-13T15:54:37.769 INFO:tasks.workunit.client.0.smithi112.stdout: 5 2024-09-13T15:54:37.769 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.770 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.770 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.770 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.770 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.770 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.770 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.770 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.770 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.770 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj3", 2024-09-13T15:54:37.770 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.770 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.770 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:54:37.770 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 44 2024-09-13T15:54:37.770 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.771 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.771 INFO:tasks.workunit.client.0.smithi112.stdout: "size_mismatch" 2024-09-13T15:54:37.771 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.771 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [ 2024-09-13T15:54:37.771 INFO:tasks.workunit.client.0.smithi112.stdout: "size_mismatch_info", 2024-09-13T15:54:37.771 INFO:tasks.workunit.client.0.smithi112.stdout: "obj_size_info_mismatch" 2024-09-13T15:54:37.771 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.771 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.771 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.771 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj3", 2024-09-13T15:54:37.771 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.771 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:54:37.771 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1643547569, 2024-09-13T15:54:37.771 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.772 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.772 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.772 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.772 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "19'44", 2024-09-13T15:54:37.772 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "17'22", 2024-09-13T15:54:37.772 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4359.0:1", 2024-09-13T15:54:37.772 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 44, 2024-09-13T15:54:37.772 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 768, 2024-09-13T15:54:37.772 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:15.402995+0000", 2024-09-13T15:54:37.772 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:15.403693+0000", 2024-09-13T15:54:37.772 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.772 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.772 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.772 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.773 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.773 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.773 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.773 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x7c506f74", 2024-09-13T15:54:37.773 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.773 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.773 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.773 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.773 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.773 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.773 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.773 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.773 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.773 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.774 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.774 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.774 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.774 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.774 INFO:tasks.workunit.client.0.smithi112.stdout: "size_mismatch_info", 2024-09-13T15:54:37.774 INFO:tasks.workunit.client.0.smithi112.stdout: "obj_size_info_mismatch" 2024-09-13T15:54:37.774 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.774 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 3840, 2024-09-13T15:54:37.774 INFO:tasks.workunit.client.0.smithi112.stdout: "object_info": { 2024-09-13T15:54:37.774 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.774 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj3", 2024-09-13T15:54:37.774 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.774 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:54:37.774 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1643547569, 2024-09-13T15:54:37.775 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.775 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.775 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.775 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.775 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "19'44", 2024-09-13T15:54:37.775 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "17'22", 2024-09-13T15:54:37.775 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4359.0:1", 2024-09-13T15:54:37.775 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 44, 2024-09-13T15:54:37.775 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 768, 2024-09-13T15:54:37.775 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:15.402995+0000", 2024-09-13T15:54:37.775 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:15.403693+0000", 2024-09-13T15:54:37.775 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.775 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.775 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.775 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.776 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.776 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.776 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.776 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x7c506f74", 2024-09-13T15:54:37.776 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.776 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.776 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.776 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.776 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.776 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.776 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.776 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.776 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.776 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.777 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.777 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.777 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.777 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.777 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 768 2024-09-13T15:54:37.777 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.777 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.777 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.777 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.777 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.777 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj6", 2024-09-13T15:54:37.777 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.777 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.777 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:54:37.778 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 24 2024-09-13T15:54:37.778 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.778 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.778 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:54:37.778 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.778 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:54:37.778 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.778 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.778 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj6", 2024-09-13T15:54:37.778 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.778 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:54:37.778 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 2202164420, 2024-09-13T15:54:37.778 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.778 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.779 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.779 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.779 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'24", 2024-09-13T15:54:37.779 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'6", 2024-09-13T15:54:37.779 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4323.0:1", 2024-09-13T15:54:37.779 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 24, 2024-09-13T15:54:37.779 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.779 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:12.518888+0000", 2024-09-13T15:54:37.779 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:12.519633+0000", 2024-09-13T15:54:37.779 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.779 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.779 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.779 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.779 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.779 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.780 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.780 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x4924c9ab", 2024-09-13T15:54:37.780 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.780 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.780 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.780 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.780 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.780 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.780 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.780 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.780 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.780 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.780 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.781 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.781 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.781 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.781 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.781 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.781 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:54:37.781 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [] 2024-09-13T15:54:37.781 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.781 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.781 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.781 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.781 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.781 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.781 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.782 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.782 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:54:37.782 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:54:37.782 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.782 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:54:37.782 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.782 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:54:37.782 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:54:37.782 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:54:37.782 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.782 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.782 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.782 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.782 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.782 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.783 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.783 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.783 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.783 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj7", 2024-09-13T15:54:37.783 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.783 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.783 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:54:37.783 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 26 2024-09-13T15:54:37.783 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.783 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.783 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:54:37.783 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.783 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:54:37.783 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.784 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.784 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj7", 2024-09-13T15:54:37.784 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.784 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:54:37.784 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 1552453721, 2024-09-13T15:54:37.784 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.784 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.784 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.784 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.784 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'26", 2024-09-13T15:54:37.784 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'7", 2024-09-13T15:54:37.784 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4326.0:1", 2024-09-13T15:54:37.784 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 26, 2024-09-13T15:54:37.784 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.785 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:12.709900+0000", 2024-09-13T15:54:37.785 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:12.710648+0000", 2024-09-13T15:54:37.785 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.785 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.785 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.785 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.785 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.785 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.785 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.785 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x4924c9ab", 2024-09-13T15:54:37.785 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.785 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.785 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.785 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.786 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.786 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.786 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.786 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.786 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.786 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.786 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.786 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.786 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.786 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.786 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.786 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.786 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 0, 2024-09-13T15:54:37.786 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [] 2024-09-13T15:54:37.787 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.787 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.787 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.787 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.787 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.787 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.787 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.787 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.787 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:54:37.787 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:54:37.787 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.787 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:54:37.787 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.787 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:54:37.788 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:54:37.788 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:54:37.788 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.788 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.788 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.788 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.788 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.788 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.788 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.788 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.788 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.788 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj8", 2024-09-13T15:54:37.788 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.788 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.789 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:54:37.789 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 28 2024-09-13T15:54:37.789 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.789 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.789 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:54:37.789 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.789 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:54:37.789 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.789 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.789 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj8", 2024-09-13T15:54:37.789 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.789 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:54:37.789 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 2381834917, 2024-09-13T15:54:37.789 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.789 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.790 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.790 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.790 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'28", 2024-09-13T15:54:37.790 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'8", 2024-09-13T15:54:37.790 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4329.0:1", 2024-09-13T15:54:37.790 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 28, 2024-09-13T15:54:37.790 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.790 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:12.911038+0000", 2024-09-13T15:54:37.790 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:12.911762+0000", 2024-09-13T15:54:37.790 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.790 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.790 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.790 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.790 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.791 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.791 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.791 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x4924c9ab", 2024-09-13T15:54:37.791 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.791 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.791 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.791 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.791 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.791 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.791 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.791 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.791 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.791 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.791 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.792 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.792 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.792 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.792 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.792 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.792 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 0, 2024-09-13T15:54:37.792 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:54:37.792 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.792 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:54:37.792 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.792 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:54:37.792 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:54:37.792 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:54:37.793 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.793 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.793 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.793 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.793 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.793 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.793 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.793 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.793 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.794 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.794 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.794 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:54:37.794 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:54:37.794 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.794 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:54:37.794 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.794 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:54:37.794 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:54:37.794 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:54:37.794 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.794 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.794 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.794 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.794 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.795 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.795 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.795 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.795 INFO:tasks.workunit.client.0.smithi112.stdout: "object": { 2024-09-13T15:54:37.795 INFO:tasks.workunit.client.0.smithi112.stdout: "name": "obj9", 2024-09-13T15:54:37.795 INFO:tasks.workunit.client.0.smithi112.stdout: "nspace": "", 2024-09-13T15:54:37.795 INFO:tasks.workunit.client.0.smithi112.stdout: "locator": "", 2024-09-13T15:54:37.795 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": "head", 2024-09-13T15:54:37.795 INFO:tasks.workunit.client.0.smithi112.stdout: "version": 30 2024-09-13T15:54:37.795 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.795 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [ 2024-09-13T15:54:37.795 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset_inconsistency" 2024-09-13T15:54:37.795 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.796 INFO:tasks.workunit.client.0.smithi112.stdout: "union_shard_errors": [], 2024-09-13T15:54:37.796 INFO:tasks.workunit.client.0.smithi112.stdout: "selected_object_info": { 2024-09-13T15:54:37.796 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": { 2024-09-13T15:54:37.796 INFO:tasks.workunit.client.0.smithi112.stdout: "oid": "obj9", 2024-09-13T15:54:37.796 INFO:tasks.workunit.client.0.smithi112.stdout: "key": "", 2024-09-13T15:54:37.796 INFO:tasks.workunit.client.0.smithi112.stdout: "snapid": -2, 2024-09-13T15:54:37.796 INFO:tasks.workunit.client.0.smithi112.stdout: "hash": 3833113727, 2024-09-13T15:54:37.796 INFO:tasks.workunit.client.0.smithi112.stdout: "max": 0, 2024-09-13T15:54:37.796 INFO:tasks.workunit.client.0.smithi112.stdout: "pool": 1, 2024-09-13T15:54:37.796 INFO:tasks.workunit.client.0.smithi112.stdout: "namespace": "" 2024-09-13T15:54:37.796 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.796 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "17'30", 2024-09-13T15:54:37.796 INFO:tasks.workunit.client.0.smithi112.stdout: "prior_version": "16'9", 2024-09-13T15:54:37.796 INFO:tasks.workunit.client.0.smithi112.stdout: "last_reqid": "client.4332.0:1", 2024-09-13T15:54:37.797 INFO:tasks.workunit.client.0.smithi112.stdout: "user_version": 30, 2024-09-13T15:54:37.797 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.797 INFO:tasks.workunit.client.0.smithi112.stdout: "mtime": "2024-09-13T15:53:13.118679+0000", 2024-09-13T15:54:37.797 INFO:tasks.workunit.client.0.smithi112.stdout: "local_mtime": "2024-09-13T15:53:13.119408+0000", 2024-09-13T15:54:37.797 INFO:tasks.workunit.client.0.smithi112.stdout: "lost": 0, 2024-09-13T15:54:37.797 INFO:tasks.workunit.client.0.smithi112.stdout: "flags": [ 2024-09-13T15:54:37.797 INFO:tasks.workunit.client.0.smithi112.stdout: "dirty", 2024-09-13T15:54:37.797 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest" 2024-09-13T15:54:37.797 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T15:54:37.797 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_seq": 0, 2024-09-13T15:54:37.797 INFO:tasks.workunit.client.0.smithi112.stdout: "truncate_size": 0, 2024-09-13T15:54:37.797 INFO:tasks.workunit.client.0.smithi112.stdout: "data_digest": "0x4924c9ab", 2024-09-13T15:54:37.797 INFO:tasks.workunit.client.0.smithi112.stdout: "omap_digest": "0xffffffff", 2024-09-13T15:54:37.797 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_object_size": 0, 2024-09-13T15:54:37.798 INFO:tasks.workunit.client.0.smithi112.stdout: "expected_write_size": 0, 2024-09-13T15:54:37.798 INFO:tasks.workunit.client.0.smithi112.stdout: "alloc_hint_flags": 0, 2024-09-13T15:54:37.798 INFO:tasks.workunit.client.0.smithi112.stdout: "manifest": { 2024-09-13T15:54:37.798 INFO:tasks.workunit.client.0.smithi112.stdout: "type": 0 2024-09-13T15:54:37.798 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.798 INFO:tasks.workunit.client.0.smithi112.stdout: "watchers": {} 2024-09-13T15:54:37.798 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.798 INFO:tasks.workunit.client.0.smithi112.stdout: "shards": [ 2024-09-13T15:54:37.798 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.798 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 0, 2024-09-13T15:54:37.798 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": false, 2024-09-13T15:54:37.798 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.798 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.798 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.799 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:54:37.799 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:54:37.799 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.799 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:54:37.799 INFO:tasks.workunit.client.0.smithi112.stdout: "size": "????", 2024-09-13T15:54:37.799 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:54:37.799 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:54:37.799 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:54:37.799 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.799 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.799 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.799 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.799 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T15:54:37.799 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.800 INFO:tasks.workunit.client.0.smithi112.stdout: "osd": 1, 2024-09-13T15:54:37.800 INFO:tasks.workunit.client.0.smithi112.stdout: "primary": true, 2024-09-13T15:54:37.800 INFO:tasks.workunit.client.0.smithi112.stdout: "errors": [], 2024-09-13T15:54:37.800 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 256, 2024-09-13T15:54:37.800 INFO:tasks.workunit.client.0.smithi112.stdout: "snapset": { 2024-09-13T15:54:37.800 INFO:tasks.workunit.client.0.smithi112.stdout: "seq": 1, 2024-09-13T15:54:37.800 INFO:tasks.workunit.client.0.smithi112.stdout: "clones": [ 2024-09-13T15:54:37.800 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T15:54:37.800 INFO:tasks.workunit.client.0.smithi112.stdout: "snap": 1, 2024-09-13T15:54:37.800 INFO:tasks.workunit.client.0.smithi112.stdout: "size": 1032, 2024-09-13T15:54:37.801 INFO:tasks.workunit.client.0.smithi112.stdout: "overlap": "[]", 2024-09-13T15:54:37.801 INFO:tasks.workunit.client.0.smithi112.stdout: "snaps": [ 2024-09-13T15:54:37.801 INFO:tasks.workunit.client.0.smithi112.stdout: 1 2024-09-13T15:54:37.801 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.801 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.801 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.801 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.801 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.801 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.801 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T15:54:37.801 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T15:54:37.801 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:54:37.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2024-09-13T15:54:37.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: '[' replica = replica ']' 2024-09-13T15:54:37.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:802: _scrub_snaps_multi: scruberrors=20 2024-09-13T15:54:37.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:803: _scrub_snaps_multi: jq .inconsistents 2024-09-13T15:54:37.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:803: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-13T15:54:37.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2024-09-13T15:54:37.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-09-13T15:54:37.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1067: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-13T15:54:37.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-09-13T15:54:37.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: test no = yes 2024-09-13T15:54:37.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1073: _scrub_snaps_multi: test '' = yes 2024-09-13T15:54:37.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: find td/osd-scrub-snaps 2024-09-13T15:54:37.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2024-09-13T15:54:37.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2024-09-13T15:54:37.805 INFO:tasks.workunit.client.0.smithi112.stderr:td/osd-scrub-snaps/osd.0.pid' 2024-09-13T15:54:37.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids= 2024-09-13T15:54:37.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-13T15:54:37.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2024-09-13T15:54:37.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='526787 ' 2024-09-13T15:54:37.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-09-13T15:54:37.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2024-09-13T15:54:37.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='526213 ' 2024-09-13T15:54:37.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1085: _scrub_snaps_multi: ERRORS=0 2024-09-13T15:54:37.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1089: _scrub_snaps_multi: '[' replica = primary ']' 2024-09-13T15:54:37.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: ceph pg dump pgs 2024-09-13T15:54:38.086 INFO:tasks.workunit.client.0.smithi112.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-13T15:54:38.086 INFO:tasks.workunit.client.0.smithi112.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent 2024-09-13T15:54:33.181400+0000 23'56 30:103 [1,0] 1 [1,0] 1 23'56 2024-09-13T15:54:33.181331+0000 0'0 2024-09-13T15:52:59.451385+0000 0 2 periodic scrub scheduled @ 2024-09-14T22:32:31.941965+0000 36 0 2024-09-13T15:54:38.086 INFO:tasks.workunit.client.0.smithi112.stdout: 2024-09-13T15:54:38.086 INFO:tasks.workunit.client.0.smithi112.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-13T15:54:38.087 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:54:38.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-09-13T15:54:38.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 526787 2024-09-13T15:54:38.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2024-09-13T15:54:38.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 526213 2024-09-13T15:54:38.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2024-09-13T15:54:38.103 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:54:38.103 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:54:38.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:54:38.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:54:38.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:54:43.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:54:43.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: declare -a err_strings 2024-09-13T15:54:43.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2024-09-13T15:54:43.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2024-09-13T15:54:43.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2024-09-13T15:54:43.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2024-09-13T15:54:43.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2024-09-13T15:54:43.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2024-09-13T15:54:43.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2024-09-13T15:54:43.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2024-09-13T15:54:43.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:54:43.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:54:43.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:54:43.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:54:43.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:54:43.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:54:43.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:54:43.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:54:43.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:54:43.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:54:43.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:54:43.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:54:43.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:54:43.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:54:43.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-09-13T15:54:43.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.1.log 2024-09-13T15:54:43.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: declare -a rep_err_strings 2024-09-13T15:54:43.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: eval echo '$replica' 2024-09-13T15:54:43.450 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 0 2024-09-13T15:54:43.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: osd=0 2024-09-13T15:54:43.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-09-13T15:54:43.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1147: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2024-09-13T15:54:43.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1149: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2024-09-13T15:54:43.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1156: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2024-09-13T15:54:43.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: _scrub_snaps_multi: echo 'TEST PASSED' 2024-09-13T15:54:43.455 INFO:tasks.workunit.client.0.smithi112.stdout:TEST PASSED 2024-09-13T15:54:43.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: _scrub_snaps_multi: return 0 2024-09-13T15:54:43.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1171: TEST_scrub_snaps_replica: err=0 2024-09-13T15:54:43.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1172: TEST_scrub_snaps_replica: CEPH_ARGS='--fsid=1fc33e19-058b-4ad9-9263-3d951a7f3d81 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-09-13T15:54:43.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1173: TEST_scrub_snaps_replica: return 0 2024-09-13T15:54:43.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-09-13T15:54:43.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-13T15:54:43.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:54:43.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-13T15:54:43.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:54:43.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:54:43.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:54:43.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:54:43.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:54:43.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:54:43.466 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:54:43.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:54:43.467 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:54:43.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:54:43.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:54:43.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:54:43.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:54:43.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:54:43.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:54:43.470 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:54:43.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:54:43.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:54:43.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-13T15:54:43.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:54:43.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:54:43.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:54:43.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.471165 2024-09-13T15:54:43.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:54:43.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:54:43.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-13T15:54:43.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-scrub-snaps 0 2024-09-13T15:54:43.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2024-09-13T15:54:43.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-13T15:54:43.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-09-13T15:54:43.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:54:43.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:54:43.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:54:43.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:54:43.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:54:43.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:54:43.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:54:43.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:54:43.497 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:54:43.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:54:43.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:54:43.498 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:54:43.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:54:43.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:54:43.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:54:43.500 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:54:43.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:54:43.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-13T15:54:43.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2024-09-13T15:54:43.503 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:54:43.503 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:54:43.503 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.471165 2024-09-13T15:54:43.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.471165 2024-09-13T15:54:43.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:54:43.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:54:43.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-13T15:54:43.505 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T15:54:43.505 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T15:54:43.558 INFO:tasks.workunit:Running workunit scrub/osd-scrub-test.sh... 2024-09-13T15:54:43.558 DEBUG:teuthology.orchestra.run.smithi112:workunit test scrub/osd-scrub-test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d58f73069a8157a9ace12a0247ee3e8497707ea9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh 2024-09-13T15:54:43.612 INFO:tasks.workunit.client.0.smithi112.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-13T15:54:43.619 INFO:tasks.workunit.client.0.smithi112.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-13T15:54:43.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-13T15:54:43.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-13T15:54:43.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-13T15:54:43.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-13T15:54:43.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-13T15:54:43.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-13T15:54:43.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-13T15:54:43.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-13T15:54:43.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-scrub-test 2024-09-13T15:54:43.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:21: run: local dir=td/osd-scrub-test 2024-09-13T15:54:43.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:22: run: shift 2024-09-13T15:54:43.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: export CEPH_MON=127.0.0.1:7138 2024-09-13T15:54:43.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: CEPH_MON=127.0.0.1:7138 2024-09-13T15:54:43.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:25: run: export CEPH_ARGS 2024-09-13T15:54:43.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: uuidgen 2024-09-13T15:54:43.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: CEPH_ARGS+='--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none ' 2024-09-13T15:54:43.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7138 ' 2024-09-13T15:54:43.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-13T15:54:43.621 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: set 2024-09-13T15:54:43.621 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-13T15:54:43.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: local 'funcs=TEST_deep_scrub_abort 2024-09-13T15:54:43.623 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_dump_scrub_schedule 2024-09-13T15:54:43.623 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_interval_changes 2024-09-13T15:54:43.623 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_just_deep_scrubs 2024-09-13T15:54:43.623 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_pg_dump_objects_scrubbed 2024-09-13T15:54:43.624 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_scrub_abort 2024-09-13T15:54:43.624 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_scrub_permit_time 2024-09-13T15:54:43.624 INFO:tasks.workunit.client.0.smithi112.stderr:TEST_scrub_test' 2024-09-13T15:54:43.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-13T15:54:43.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_deep_scrub_abort -------------------' 2024-09-13T15:54:43.624 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Prepare Test TEST_deep_scrub_abort ------------------- 2024-09-13T15:54:43.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-09-13T15:54:43.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2024-09-13T15:54:43.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2024-09-13T15:54:43.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T15:54:43.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T15:54:43.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T15:54:43.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T15:54:43.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T15:54:43.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T15:54:43.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T15:54:43.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T15:54:43.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T15:54:43.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T15:54:43.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T15:54:43.628 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T15:54:43.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T15:54:43.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T15:54:43.629 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T15:54:43.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:54:43.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T15:54:43.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T15:54:43.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:54:43.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T15:54:43.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T15:54:43.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T15:54:43.634 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T15:54:43.634 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:54:43.634 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T15:54:43.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T15:54:43.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T15:54:43.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T15:54:43.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2024-09-13T15:54:43.636 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T15:54:43.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:54:43.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T15:54:43.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.528094 2024-09-13T15:54:43.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T15:54:43.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-13T15:54:43.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-13T15:54:43.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T15:54:43.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-13T15:54:43.640 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Run Test TEST_deep_scrub_abort ----------------------- 2024-09-13T15:54:43.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_deep_scrub_abort -----------------------' 2024-09-13T15:54:43.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_deep_scrub_abort td/osd-scrub-test 2024-09-13T15:54:43.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:416: TEST_deep_scrub_abort: local dir=td/osd-scrub-test 2024-09-13T15:54:43.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:417: TEST_deep_scrub_abort: _scrub_abort td/osd-scrub-test deep-scrub 2024-09-13T15:54:43.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: local dir=td/osd-scrub-test 2024-09-13T15:54:43.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:301: _scrub_abort: local poolname=test 2024-09-13T15:54:43.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:302: _scrub_abort: local OSDS=3 2024-09-13T15:54:43.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:303: _scrub_abort: local objects=1000 2024-09-13T15:54:43.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:304: _scrub_abort: local type=deep-scrub 2024-09-13T15:54:43.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:306: _scrub_abort: TESTDATA=testdata.528094 2024-09-13T15:54:43.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:307: _scrub_abort: test deep-scrub = scrub 2024-09-13T15:54:43.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:312: _scrub_abort: stopscrub=nodeep-scrub 2024-09-13T15:54:43.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:313: _scrub_abort: check=nodeep_scrub 2024-09-13T15:54:43.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-13T15:54:43.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2024-09-13T15:54:43.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T15:54:43.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T15:54:43.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T15:54:43.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2024-09-13T15:54:43.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-13T15:54:44.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T15:54:44.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:54:44.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:54:44.170 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:54:44.170 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:54:44.170 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T15:54:44.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T15:54:44.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-13T15:54:44.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T15:54:44.415 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T15:54:44.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:54:44.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:54:44.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T15:54:44.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T15:54:44.417 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:54:44.417 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:54:44.417 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:54:44.417 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:54:44.417 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:54:44.417 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T15:54:44.418 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-mon.a.asok 2024-09-13T15:54:44.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:54:44.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.528094/ceph-mon.a.asok config get fsid 2024-09-13T15:54:44.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T15:54:44.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T15:54:44.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T15:54:44.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T15:54:44.536 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T15:54:44.536 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T15:54:44.536 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T15:54:44.536 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T15:54:44.536 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T15:54:44.536 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:54:44.536 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T15:54:44.536 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-mon.a.asok 2024-09-13T15:54:44.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T15:54:44.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.528094/ceph-mon.a.asok config get mon_host 2024-09-13T15:54:44.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_mgr td/osd-scrub-test x 2024-09-13T15:54:44.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2024-09-13T15:54:44.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T15:54:44.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T15:54:44.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T15:54:44.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2024-09-13T15:54:44.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T15:54:45.550 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T15:54:45.550 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:54:45.550 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:54:45.550 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:54:45.550 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:54:45.551 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T15:54:45.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T15:54:45.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:54:45.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T15:54:45.586 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: expr 3 - 1 2024-09-13T15:54:45.587 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: seq 0 2 2024-09-13T15:54:45.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:54:45.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:324: _scrub_abort: run_osd td/osd-scrub-test 0 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-13T15:54:45.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T15:54:45.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:54:45.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T15:54:45.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:54:45.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-13T15:54:45.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T15:54:45.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:54:45.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:54:45.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:54:45.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-13T15:54:45.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-13T15:54:45.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:54:45.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:54:45.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T15:54:45.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:54:45.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:54:45.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:54:45.592 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:54:45.592 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:54:45.592 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T15:54:45.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T15:54:45.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T15:54:45.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:54:45.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:54:45.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:54:45.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T15:54:45.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T15:54:45.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:54:45.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:54:45.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:54:45.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:54:45.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:54:45.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-13T15:54:45.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-13T15:54:45.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:54:45.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f2730e8f-6c2c-477a-a53a-c74ee00eb2cf 2024-09-13T15:54:45.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f2730e8f-6c2c-477a-a53a-c74ee00eb2cf' 2024-09-13T15:54:45.595 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 f2730e8f-6c2c-477a-a53a-c74ee00eb2cf 2024-09-13T15:54:45.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:54:45.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBFYORmUTFCJBAAQgiekf8VA2u9Tys7JVmqrg== 2024-09-13T15:54:45.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBFYORmUTFCJBAAQgiekf8VA2u9Tys7JVmqrg=="}' 2024-09-13T15:54:45.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f2730e8f-6c2c-477a-a53a-c74ee00eb2cf -i td/osd-scrub-test/0/new.json 2024-09-13T15:54:45.892 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:54:45.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-13T15:54:45.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQBFYORmUTFCJBAAQgiekf8VA2u9Tys7JVmqrg== --osd-uuid f2730e8f-6c2c-477a-a53a-c74ee00eb2cf 2024-09-13T15:54:45.947 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:45.944+0000 7f4a3aecc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:54:45.976 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:45.976+0000 7f4a3aecc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:54:45.978 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:45.976+0000 7f4a3aecc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:54:45.978 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:45.976+0000 7f4a3aecc7c0 -1 bdev(0x55ecad228800 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2024-09-13T15:54:45.978 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:45.976+0000 7f4a3aecc7c0 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-13T15:54:54.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-13T15:54:54.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:54:54.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T15:54:54.132 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T15:54:54.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:54:54.540 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T15:54:54.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T15:54:54.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-13T15:54:54.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:54:54.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:54:54.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:54:54.573 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:54.571+0000 7fbdfdac77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:54:54.616 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:54.615+0000 7fbdfdac77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:54:54.617 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:54.615+0000 7fbdfdac77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:54:54.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T15:54:54.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:54:54.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T15:54:54.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:54:54.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:54:54.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:54:54.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:54:54.850 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:54:54.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:54:54.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:54:55.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:54:55.686 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:55.683+0000 7fbdfdac77c0 -1 Falling back to public interface 2024-09-13T15:54:56.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:54:56.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:54:56.229 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:54:56.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:54:56.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:54:56.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:54:56.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:54:57.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:54:57.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:54:57.543 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:54:57.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:54:57.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:54:57.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:54:57.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:54:58.821 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:54:58.819+0000 7fbdfdac77c0 -1 osd.0 0 log_to_monitors true 2024-09-13T15:54:58.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:54:58.865 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:54:58.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:54:58.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:54:58.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:54:58.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:54:59.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:55:00.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:55:00.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:55:00.333 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:55:00.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:55:00.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:55:00.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T15:55:00.676 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3089297226,v1:127.0.0.1:6803/3089297226] [v2:127.0.0.1:6804/3089297226,v1:127.0.0.1:6805/3089297226] exists,up f2730e8f-6c2c-477a-a53a-c74ee00eb2cf 2024-09-13T15:55:00.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:55:00.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:55:00.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:55:00.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:55:00.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:324: _scrub_abort: run_osd td/osd-scrub-test 1 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-13T15:55:00.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T15:55:00.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:55:00.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T15:55:00.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:55:00.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-13T15:55:00.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T15:55:00.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:55:00.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:55:00.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:55:00.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-13T15:55:00.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-13T15:55:00.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:55:00.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:55:00.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T15:55:00.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:55:00.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:55:00.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:55:00.682 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:55:00.682 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:55:00.682 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T15:55:00.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T15:55:00.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T15:55:00.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:55:00.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:55:00.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:55:00.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T15:55:00.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T15:55:00.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:55:00.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:55:00.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:55:00.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:55:00.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:55:00.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-13T15:55:00.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-13T15:55:00.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:55:00.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=03244948-0544-4503-ba0e-29ab52bb6832 2024-09-13T15:55:00.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 03244948-0544-4503-ba0e-29ab52bb6832' 2024-09-13T15:55:00.685 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 03244948-0544-4503-ba0e-29ab52bb6832 2024-09-13T15:55:00.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:55:00.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBUYORmEg+0KRAAXB1P/OVI6uVyCDYpcuSpkA== 2024-09-13T15:55:00.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBUYORmEg+0KRAAXB1P/OVI6uVyCDYpcuSpkA=="}' 2024-09-13T15:55:00.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 03244948-0544-4503-ba0e-29ab52bb6832 -i td/osd-scrub-test/1/new.json 2024-09-13T15:55:01.227 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:55:01.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-13T15:55:01.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQBUYORmEg+0KRAAXB1P/OVI6uVyCDYpcuSpkA== --osd-uuid 03244948-0544-4503-ba0e-29ab52bb6832 2024-09-13T15:55:01.285 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:01.283+0000 7f1612d3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:55:01.371 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:01.371+0000 7f1612d3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:55:01.373 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:01.371+0000 7f1612d3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:55:01.373 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:01.371+0000 7f1612d3b7c0 -1 bdev(0x5647c0a1bc00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2024-09-13T15:55:01.374 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:01.371+0000 7f1612d3b7c0 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-13T15:55:09.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-13T15:55:09.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:55:09.781 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T15:55:09.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T15:55:09.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:55:10.188 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T15:55:10.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T15:55:10.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:55:10.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-13T15:55:10.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:55:10.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:55:10.221 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:10.219+0000 7fe29870b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:55:10.267 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:10.267+0000 7fe29870b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:55:10.269 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:10.267+0000 7fe29870b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:55:10.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T15:55:10.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:55:10.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T15:55:10.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:55:10.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:55:10.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:55:10.510 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:55:10.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:55:10.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:55:10.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:55:10.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:55:11.275 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:11.271+0000 7fe29870b7c0 -1 Falling back to public interface 2024-09-13T15:55:11.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:55:11.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:55:11.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:55:11.834 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:55:11.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:55:11.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:55:12.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:55:13.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:55:13.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:55:13.165 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:55:13.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:55:13.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:55:13.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:55:13.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:55:14.355 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:14.355+0000 7fe29870b7c0 -1 osd.1 0 log_to_monitors true 2024-09-13T15:55:14.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:55:14.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:55:14.479 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:55:14.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:55:14.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:55:14.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:55:14.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:55:15.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:55:15.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:55:15.969 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:55:15.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:55:15.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:55:15.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T15:55:16.285 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3799985864,v1:127.0.0.1:6811/3799985864] [v2:127.0.0.1:6812/3799985864,v1:127.0.0.1:6813/3799985864] exists,up 03244948-0544-4503-ba0e-29ab52bb6832 2024-09-13T15:55:16.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:55:16.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:55:16.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:55:16.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T15:55:16.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:324: _scrub_abort: run_osd td/osd-scrub-test 2 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-13T15:55:16.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T15:55:16.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T15:55:16.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T15:55:16.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T15:55:16.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-13T15:55:16.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T15:55:16.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T15:55:16.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T15:55:16.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T15:55:16.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-13T15:55:16.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-13T15:55:16.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T15:55:16.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T15:55:16.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T15:55:16.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T15:55:16.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T15:55:16.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T15:55:16.290 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T15:55:16.290 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T15:55:16.290 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T15:55:16.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T15:55:16.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T15:55:16.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T15:55:16.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T15:55:16.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T15:55:16.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T15:55:16.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T15:55:16.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T15:55:16.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T15:55:16.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T15:55:16.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T15:55:16.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T15:55:16.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-13T15:55:16.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-13T15:55:16.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T15:55:16.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7250830b-d1fd-4c23-8ca2-26fd9a13467c 2024-09-13T15:55:16.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 7250830b-d1fd-4c23-8ca2-26fd9a13467c' 2024-09-13T15:55:16.293 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 7250830b-d1fd-4c23-8ca2-26fd9a13467c 2024-09-13T15:55:16.293 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T15:55:16.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBkYORmVKFSEhAAXdwSkVzNM5LJPoqCqWchXQ== 2024-09-13T15:55:16.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBkYORmVKFSEhAAXdwSkVzNM5LJPoqCqWchXQ=="}' 2024-09-13T15:55:16.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7250830b-d1fd-4c23-8ca2-26fd9a13467c -i td/osd-scrub-test/2/new.json 2024-09-13T15:55:16.956 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:55:16.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-13T15:55:16.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQBkYORmVKFSEhAAXdwSkVzNM5LJPoqCqWchXQ== --osd-uuid 7250830b-d1fd-4c23-8ca2-26fd9a13467c 2024-09-13T15:55:17.014 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:17.011+0000 7f6fb1d9a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:55:17.091 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:17.091+0000 7f6fb1d9a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:55:17.093 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:17.091+0000 7f6fb1d9a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:55:17.094 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:17.091+0000 7f6fb1d9a7c0 -1 bdev(0x55ec9aa69c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2024-09-13T15:55:17.094 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:17.091+0000 7f6fb1d9a7c0 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-13T15:55:25.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-13T15:55:25.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T15:55:25.463 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T15:55:25.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T15:55:25.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T15:55:25.868 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T15:55:25.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T15:55:25.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T15:55:25.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-13T15:55:25.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T15:55:25.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T15:55:25.903 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:25.903+0000 7f8d39cda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:55:25.938 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:25.935+0000 7f8d39cda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:55:25.939 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:25.939+0000 7f8d39cda7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T15:55:26.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T15:55:26.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T15:55:26.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T15:55:26.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T15:55:26.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T15:55:26.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:55:26.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T15:55:26.164 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T15:55:26.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:55:26.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:55:26.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:55:26.982 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:26.979+0000 7f8d39cda7c0 -1 Falling back to public interface 2024-09-13T15:55:27.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:55:27.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:55:27.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T15:55:27.490 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T15:55:27.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:55:27.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:55:27.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:55:28.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:55:28.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:55:28.806 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T15:55:28.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T15:55:28.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:55:28.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:55:29.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:55:30.026 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T15:55:30.023+0000 7f8d39cda7c0 -1 osd.2 0 log_to_monitors true 2024-09-13T15:55:30.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:55:30.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:55:30.141 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T15:55:30.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T15:55:30.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:55:30.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:55:30.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T15:55:31.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T15:55:31.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T15:55:31.663 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T15:55:31.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T15:55:31.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T15:55:31.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T15:55:31.989 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1669422419,v1:127.0.0.1:6819/1669422419] [v2:127.0.0.1:6820/1669422419,v1:127.0.0.1:6821/1669422419] exists,up 7250830b-d1fd-4c23-8ca2-26fd9a13467c 2024-09-13T15:55:31.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T15:55:31.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T15:55:31.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T15:55:31.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: create_pool test 1 1 2024-09-13T15:55:31.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-13T15:55:32.777 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'test' created 2024-09-13T15:55:32.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T15:55:33.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:333: _scrub_abort: wait_for_clean 2024-09-13T15:55:33.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T15:55:33.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T15:55:33.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T15:55:33.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T15:55:33.803 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T15:55:33.803 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T15:55:33.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T15:55:33.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T15:55:33.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T15:55:33.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T15:55:33.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T15:55:33.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T15:55:33.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T15:55:33.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:55:33.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:55:34.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:55:34.255 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:55:34.255 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:55:34.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:55:34.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:55:34.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:55:34.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836489 2024-09-13T15:55:34.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836489 2024-09-13T15:55:34.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489' 2024-09-13T15:55:34.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:55:34.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:55:34.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-13T15:55:34.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-13T15:55:34.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489 1-42949672965' 2024-09-13T15:55:34.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:55:34.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:55:34.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509442 2024-09-13T15:55:34.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509442 2024-09-13T15:55:34.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489 1-42949672965 2-64424509442' 2024-09-13T15:55:34.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:55:34.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836489 2024-09-13T15:55:34.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:55:34.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:55:34.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836489 2024-09-13T15:55:34.701 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:55:34.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836489 2024-09-13T15:55:34.702 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836489 2024-09-13T15:55:34.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-09-13T15:55:34.703 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:55:35.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836489 2024-09-13T15:55:35.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:55:36.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:55:36.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:55:36.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836489 2024-09-13T15:55:36.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:55:37.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T15:55:37.325 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:55:37.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836489 2024-09-13T15:55:37.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:55:37.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-13T15:55:37.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:55:37.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:55:37.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-13T15:55:37.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:55:37.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-13T15:55:37.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-13T15:55:37.659 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672965 2024-09-13T15:55:37.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:55:38.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672966 -lt 42949672965 2024-09-13T15:55:38.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:55:38.027 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509442 2024-09-13T15:55:38.027 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:55:38.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:55:38.029 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509442 2024-09-13T15:55:38.029 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:55:38.030 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509442 2024-09-13T15:55:38.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509442 2024-09-13T15:55:38.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-09-13T15:55:38.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:55:38.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509442 -lt 64424509442 2024-09-13T15:55:38.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T15:55:38.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:55:38.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:55:38.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T15:55:38.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T15:55:38.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T15:55:38.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T15:55:38.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T15:55:38.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T15:55:38.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T15:55:38.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T15:55:39.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T15:55:39.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T15:55:39.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T15:55:39.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T15:55:39.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T15:55:39.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T15:55:39.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T15:55:39.499 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: grep '^pool.*['\'']test['\'']' 2024-09-13T15:55:39.499 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: ceph osd dump 2024-09-13T15:55:39.499 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: awk '{ print $2 }' 2024-09-13T15:55:39.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: poolid=1 2024-09-13T15:55:39.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: dd if=/dev/urandom of=testdata.528094 bs=1032 count=1 2024-09-13T15:55:39.856 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T15:55:39.856 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T15:55:39.856 INFO:tasks.workunit.client.0.smithi112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000108577 s, 9.5 MB/s 2024-09-13T15:55:39.857 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: seq 1 1000 2024-09-13T15:55:39.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:39.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj1 testdata.528094 2024-09-13T15:55:40.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:40.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj2 testdata.528094 2024-09-13T15:55:40.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:40.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj3 testdata.528094 2024-09-13T15:55:40.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:40.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj4 testdata.528094 2024-09-13T15:55:40.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:40.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj5 testdata.528094 2024-09-13T15:55:40.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:40.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj6 testdata.528094 2024-09-13T15:55:41.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:41.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj7 testdata.528094 2024-09-13T15:55:41.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:41.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj8 testdata.528094 2024-09-13T15:55:41.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:41.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj9 testdata.528094 2024-09-13T15:55:41.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:41.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj10 testdata.528094 2024-09-13T15:55:42.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:42.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj11 testdata.528094 2024-09-13T15:55:42.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:42.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj12 testdata.528094 2024-09-13T15:55:42.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:42.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj13 testdata.528094 2024-09-13T15:55:42.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:42.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj14 testdata.528094 2024-09-13T15:55:42.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:42.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj15 testdata.528094 2024-09-13T15:55:43.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:43.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj16 testdata.528094 2024-09-13T15:55:43.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:43.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj17 testdata.528094 2024-09-13T15:55:43.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:43.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj18 testdata.528094 2024-09-13T15:55:43.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:43.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj19 testdata.528094 2024-09-13T15:55:43.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:43.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj20 testdata.528094 2024-09-13T15:55:44.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:44.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj21 testdata.528094 2024-09-13T15:55:44.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:44.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj22 testdata.528094 2024-09-13T15:55:44.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:44.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj23 testdata.528094 2024-09-13T15:55:44.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:44.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj24 testdata.528094 2024-09-13T15:55:44.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:44.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj25 testdata.528094 2024-09-13T15:55:45.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:45.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj26 testdata.528094 2024-09-13T15:55:45.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:45.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj27 testdata.528094 2024-09-13T15:55:45.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:45.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj28 testdata.528094 2024-09-13T15:55:45.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:45.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj29 testdata.528094 2024-09-13T15:55:45.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:45.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj30 testdata.528094 2024-09-13T15:55:46.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:46.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj31 testdata.528094 2024-09-13T15:55:46.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:46.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj32 testdata.528094 2024-09-13T15:55:46.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:46.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj33 testdata.528094 2024-09-13T15:55:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:46.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj34 testdata.528094 2024-09-13T15:55:47.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:47.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj35 testdata.528094 2024-09-13T15:55:47.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:47.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj36 testdata.528094 2024-09-13T15:55:47.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:47.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj37 testdata.528094 2024-09-13T15:55:47.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:47.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj38 testdata.528094 2024-09-13T15:55:47.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:47.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj39 testdata.528094 2024-09-13T15:55:48.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:48.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj40 testdata.528094 2024-09-13T15:55:48.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:48.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj41 testdata.528094 2024-09-13T15:55:48.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:48.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj42 testdata.528094 2024-09-13T15:55:48.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:48.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj43 testdata.528094 2024-09-13T15:55:48.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:48.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj44 testdata.528094 2024-09-13T15:55:49.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:49.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj45 testdata.528094 2024-09-13T15:55:49.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:49.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj46 testdata.528094 2024-09-13T15:55:49.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:49.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj47 testdata.528094 2024-09-13T15:55:49.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:49.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj48 testdata.528094 2024-09-13T15:55:49.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:49.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj49 testdata.528094 2024-09-13T15:55:50.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:50.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj50 testdata.528094 2024-09-13T15:55:50.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:50.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj51 testdata.528094 2024-09-13T15:55:50.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:50.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj52 testdata.528094 2024-09-13T15:55:50.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:50.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj53 testdata.528094 2024-09-13T15:55:50.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:50.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj54 testdata.528094 2024-09-13T15:55:51.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:51.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj55 testdata.528094 2024-09-13T15:55:51.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:51.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj56 testdata.528094 2024-09-13T15:55:51.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:51.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj57 testdata.528094 2024-09-13T15:55:51.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:51.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj58 testdata.528094 2024-09-13T15:55:52.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:52.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj59 testdata.528094 2024-09-13T15:55:52.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:52.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj60 testdata.528094 2024-09-13T15:55:52.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:52.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj61 testdata.528094 2024-09-13T15:55:52.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:52.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj62 testdata.528094 2024-09-13T15:55:52.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:52.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj63 testdata.528094 2024-09-13T15:55:53.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:53.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj64 testdata.528094 2024-09-13T15:55:53.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:53.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj65 testdata.528094 2024-09-13T15:55:53.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:53.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj66 testdata.528094 2024-09-13T15:55:53.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:53.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj67 testdata.528094 2024-09-13T15:55:53.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:53.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj68 testdata.528094 2024-09-13T15:55:54.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:54.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj69 testdata.528094 2024-09-13T15:55:54.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:54.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj70 testdata.528094 2024-09-13T15:55:54.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:54.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj71 testdata.528094 2024-09-13T15:55:54.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:54.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj72 testdata.528094 2024-09-13T15:55:54.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:54.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj73 testdata.528094 2024-09-13T15:55:55.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:55.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj74 testdata.528094 2024-09-13T15:55:55.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:55.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj75 testdata.528094 2024-09-13T15:55:55.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:55.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj76 testdata.528094 2024-09-13T15:55:55.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:55.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj77 testdata.528094 2024-09-13T15:55:56.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:56.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj78 testdata.528094 2024-09-13T15:55:56.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:56.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj79 testdata.528094 2024-09-13T15:55:56.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:56.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj80 testdata.528094 2024-09-13T15:55:56.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:56.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj81 testdata.528094 2024-09-13T15:55:56.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:56.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj82 testdata.528094 2024-09-13T15:55:57.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:57.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj83 testdata.528094 2024-09-13T15:55:57.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:57.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj84 testdata.528094 2024-09-13T15:55:57.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:57.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj85 testdata.528094 2024-09-13T15:55:57.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:57.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj86 testdata.528094 2024-09-13T15:55:57.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:57.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj87 testdata.528094 2024-09-13T15:55:58.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:58.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj88 testdata.528094 2024-09-13T15:55:58.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:58.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj89 testdata.528094 2024-09-13T15:55:58.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:58.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj90 testdata.528094 2024-09-13T15:55:58.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:58.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj91 testdata.528094 2024-09-13T15:55:58.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:58.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj92 testdata.528094 2024-09-13T15:55:59.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:59.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj93 testdata.528094 2024-09-13T15:55:59.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:59.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj94 testdata.528094 2024-09-13T15:55:59.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:59.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj95 testdata.528094 2024-09-13T15:55:59.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:59.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj96 testdata.528094 2024-09-13T15:55:59.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:55:59.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj97 testdata.528094 2024-09-13T15:56:00.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:00.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj98 testdata.528094 2024-09-13T15:56:00.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:00.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj99 testdata.528094 2024-09-13T15:56:00.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:00.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj100 testdata.528094 2024-09-13T15:56:00.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:00.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj101 testdata.528094 2024-09-13T15:56:01.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:01.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj102 testdata.528094 2024-09-13T15:56:01.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:01.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj103 testdata.528094 2024-09-13T15:56:01.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:01.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj104 testdata.528094 2024-09-13T15:56:01.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:01.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj105 testdata.528094 2024-09-13T15:56:01.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:01.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj106 testdata.528094 2024-09-13T15:56:02.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:02.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj107 testdata.528094 2024-09-13T15:56:02.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:02.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj108 testdata.528094 2024-09-13T15:56:02.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:02.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj109 testdata.528094 2024-09-13T15:56:02.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:02.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj110 testdata.528094 2024-09-13T15:56:02.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:02.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj111 testdata.528094 2024-09-13T15:56:03.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:03.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj112 testdata.528094 2024-09-13T15:56:03.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:03.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj113 testdata.528094 2024-09-13T15:56:03.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:03.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj114 testdata.528094 2024-09-13T15:56:04.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:04.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj115 testdata.528094 2024-09-13T15:56:04.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:04.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj116 testdata.528094 2024-09-13T15:56:04.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:04.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj117 testdata.528094 2024-09-13T15:56:04.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:04.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj118 testdata.528094 2024-09-13T15:56:04.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:04.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj119 testdata.528094 2024-09-13T15:56:05.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:05.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj120 testdata.528094 2024-09-13T15:56:05.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:05.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj121 testdata.528094 2024-09-13T15:56:05.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:05.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj122 testdata.528094 2024-09-13T15:56:05.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:05.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj123 testdata.528094 2024-09-13T15:56:05.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:05.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj124 testdata.528094 2024-09-13T15:56:06.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:06.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj125 testdata.528094 2024-09-13T15:56:06.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:06.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj126 testdata.528094 2024-09-13T15:56:06.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:06.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj127 testdata.528094 2024-09-13T15:56:06.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:06.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj128 testdata.528094 2024-09-13T15:56:07.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:07.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj129 testdata.528094 2024-09-13T15:56:07.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:07.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj130 testdata.528094 2024-09-13T15:56:07.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:07.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj131 testdata.528094 2024-09-13T15:56:07.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:07.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj132 testdata.528094 2024-09-13T15:56:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:07.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj133 testdata.528094 2024-09-13T15:56:08.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:08.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj134 testdata.528094 2024-09-13T15:56:08.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:08.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj135 testdata.528094 2024-09-13T15:56:08.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:08.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj136 testdata.528094 2024-09-13T15:56:08.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:08.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj137 testdata.528094 2024-09-13T15:56:08.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:08.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj138 testdata.528094 2024-09-13T15:56:09.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:09.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj139 testdata.528094 2024-09-13T15:56:09.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:09.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj140 testdata.528094 2024-09-13T15:56:09.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:09.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj141 testdata.528094 2024-09-13T15:56:09.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:09.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj142 testdata.528094 2024-09-13T15:56:09.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:09.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj143 testdata.528094 2024-09-13T15:56:10.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:10.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj144 testdata.528094 2024-09-13T15:56:10.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:10.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj145 testdata.528094 2024-09-13T15:56:10.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:10.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj146 testdata.528094 2024-09-13T15:56:10.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:10.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj147 testdata.528094 2024-09-13T15:56:10.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:10.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj148 testdata.528094 2024-09-13T15:56:11.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:11.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj149 testdata.528094 2024-09-13T15:56:11.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:11.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj150 testdata.528094 2024-09-13T15:56:11.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:11.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj151 testdata.528094 2024-09-13T15:56:11.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:11.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj152 testdata.528094 2024-09-13T15:56:11.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:11.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj153 testdata.528094 2024-09-13T15:56:12.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:12.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj154 testdata.528094 2024-09-13T15:56:12.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:12.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj155 testdata.528094 2024-09-13T15:56:12.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:12.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj156 testdata.528094 2024-09-13T15:56:12.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:12.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj157 testdata.528094 2024-09-13T15:56:13.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:13.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj158 testdata.528094 2024-09-13T15:56:13.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:13.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj159 testdata.528094 2024-09-13T15:56:13.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:13.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj160 testdata.528094 2024-09-13T15:56:13.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:13.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj161 testdata.528094 2024-09-13T15:56:13.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:13.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj162 testdata.528094 2024-09-13T15:56:14.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:14.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj163 testdata.528094 2024-09-13T15:56:14.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:14.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj164 testdata.528094 2024-09-13T15:56:14.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:14.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj165 testdata.528094 2024-09-13T15:56:14.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:14.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj166 testdata.528094 2024-09-13T15:56:14.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:14.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj167 testdata.528094 2024-09-13T15:56:15.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:15.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj168 testdata.528094 2024-09-13T15:56:15.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:15.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj169 testdata.528094 2024-09-13T15:56:15.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:15.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj170 testdata.528094 2024-09-13T15:56:15.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:15.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj171 testdata.528094 2024-09-13T15:56:15.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:15.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj172 testdata.528094 2024-09-13T15:56:15.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:15.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj173 testdata.528094 2024-09-13T15:56:16.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:16.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj174 testdata.528094 2024-09-13T15:56:16.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:16.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj175 testdata.528094 2024-09-13T15:56:16.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:16.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj176 testdata.528094 2024-09-13T15:56:16.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:16.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj177 testdata.528094 2024-09-13T15:56:17.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:17.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj178 testdata.528094 2024-09-13T15:56:17.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:17.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj179 testdata.528094 2024-09-13T15:56:17.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:17.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj180 testdata.528094 2024-09-13T15:56:17.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:17.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj181 testdata.528094 2024-09-13T15:56:17.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:17.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj182 testdata.528094 2024-09-13T15:56:18.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:18.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj183 testdata.528094 2024-09-13T15:56:18.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:18.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj184 testdata.528094 2024-09-13T15:56:18.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:18.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj185 testdata.528094 2024-09-13T15:56:18.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:18.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj186 testdata.528094 2024-09-13T15:56:18.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:18.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj187 testdata.528094 2024-09-13T15:56:19.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:19.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj188 testdata.528094 2024-09-13T15:56:19.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:19.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj189 testdata.528094 2024-09-13T15:56:19.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:19.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj190 testdata.528094 2024-09-13T15:56:19.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:19.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj191 testdata.528094 2024-09-13T15:56:19.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:19.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj192 testdata.528094 2024-09-13T15:56:20.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:20.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj193 testdata.528094 2024-09-13T15:56:20.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:20.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj194 testdata.528094 2024-09-13T15:56:20.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:20.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj195 testdata.528094 2024-09-13T15:56:20.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:20.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj196 testdata.528094 2024-09-13T15:56:21.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:21.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj197 testdata.528094 2024-09-13T15:56:21.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:21.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj198 testdata.528094 2024-09-13T15:56:21.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:21.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj199 testdata.528094 2024-09-13T15:56:21.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:21.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj200 testdata.528094 2024-09-13T15:56:21.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:21.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj201 testdata.528094 2024-09-13T15:56:22.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:22.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj202 testdata.528094 2024-09-13T15:56:22.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:22.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj203 testdata.528094 2024-09-13T15:56:22.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:22.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj204 testdata.528094 2024-09-13T15:56:22.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:22.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj205 testdata.528094 2024-09-13T15:56:22.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:22.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj206 testdata.528094 2024-09-13T15:56:23.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:23.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj207 testdata.528094 2024-09-13T15:56:23.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:23.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj208 testdata.528094 2024-09-13T15:56:23.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:23.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj209 testdata.528094 2024-09-13T15:56:23.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:23.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj210 testdata.528094 2024-09-13T15:56:23.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:23.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj211 testdata.528094 2024-09-13T15:56:24.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:24.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj212 testdata.528094 2024-09-13T15:56:24.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:24.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj213 testdata.528094 2024-09-13T15:56:24.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:24.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj214 testdata.528094 2024-09-13T15:56:24.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:24.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj215 testdata.528094 2024-09-13T15:56:24.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:24.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj216 testdata.528094 2024-09-13T15:56:25.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:25.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj217 testdata.528094 2024-09-13T15:56:25.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:25.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj218 testdata.528094 2024-09-13T15:56:25.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:25.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj219 testdata.528094 2024-09-13T15:56:25.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:25.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj220 testdata.528094 2024-09-13T15:56:25.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:25.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj221 testdata.528094 2024-09-13T15:56:26.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:26.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj222 testdata.528094 2024-09-13T15:56:26.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:26.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj223 testdata.528094 2024-09-13T15:56:26.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:26.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj224 testdata.528094 2024-09-13T15:56:26.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:26.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj225 testdata.528094 2024-09-13T15:56:27.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:27.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj226 testdata.528094 2024-09-13T15:56:27.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:27.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj227 testdata.528094 2024-09-13T15:56:27.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:27.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj228 testdata.528094 2024-09-13T15:56:27.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:27.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj229 testdata.528094 2024-09-13T15:56:27.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:27.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj230 testdata.528094 2024-09-13T15:56:28.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:28.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj231 testdata.528094 2024-09-13T15:56:28.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:28.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj232 testdata.528094 2024-09-13T15:56:28.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:28.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj233 testdata.528094 2024-09-13T15:56:28.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:28.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj234 testdata.528094 2024-09-13T15:56:28.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:28.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj235 testdata.528094 2024-09-13T15:56:29.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:29.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj236 testdata.528094 2024-09-13T15:56:29.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:29.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj237 testdata.528094 2024-09-13T15:56:29.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:29.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj238 testdata.528094 2024-09-13T15:56:29.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:29.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj239 testdata.528094 2024-09-13T15:56:29.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:29.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj240 testdata.528094 2024-09-13T15:56:30.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:30.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj241 testdata.528094 2024-09-13T15:56:30.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:30.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj242 testdata.528094 2024-09-13T15:56:30.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:30.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj243 testdata.528094 2024-09-13T15:56:30.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:30.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj244 testdata.528094 2024-09-13T15:56:30.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:30.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj245 testdata.528094 2024-09-13T15:56:31.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:31.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj246 testdata.528094 2024-09-13T15:56:31.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:31.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj247 testdata.528094 2024-09-13T15:56:31.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:31.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj248 testdata.528094 2024-09-13T15:56:31.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:31.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj249 testdata.528094 2024-09-13T15:56:31.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:31.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj250 testdata.528094 2024-09-13T15:56:32.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:32.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj251 testdata.528094 2024-09-13T15:56:32.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:32.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj252 testdata.528094 2024-09-13T15:56:32.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:32.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj253 testdata.528094 2024-09-13T15:56:32.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:32.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj254 testdata.528094 2024-09-13T15:56:33.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:33.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj255 testdata.528094 2024-09-13T15:56:33.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:33.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj256 testdata.528094 2024-09-13T15:56:33.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:33.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj257 testdata.528094 2024-09-13T15:56:33.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:33.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj258 testdata.528094 2024-09-13T15:56:33.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:33.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj259 testdata.528094 2024-09-13T15:56:34.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:34.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj260 testdata.528094 2024-09-13T15:56:34.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:34.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj261 testdata.528094 2024-09-13T15:56:34.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:34.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj262 testdata.528094 2024-09-13T15:56:35.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:35.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj263 testdata.528094 2024-09-13T15:56:35.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:35.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj264 testdata.528094 2024-09-13T15:56:35.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:35.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj265 testdata.528094 2024-09-13T15:56:35.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:35.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj266 testdata.528094 2024-09-13T15:56:36.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:36.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj267 testdata.528094 2024-09-13T15:56:36.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:36.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj268 testdata.528094 2024-09-13T15:56:36.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:36.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj269 testdata.528094 2024-09-13T15:56:36.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:36.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj270 testdata.528094 2024-09-13T15:56:36.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:36.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj271 testdata.528094 2024-09-13T15:56:37.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:37.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj272 testdata.528094 2024-09-13T15:56:37.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:37.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj273 testdata.528094 2024-09-13T15:56:37.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:37.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj274 testdata.528094 2024-09-13T15:56:37.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:37.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj275 testdata.528094 2024-09-13T15:56:37.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:37.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj276 testdata.528094 2024-09-13T15:56:38.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:38.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj277 testdata.528094 2024-09-13T15:56:38.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:38.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj278 testdata.528094 2024-09-13T15:56:38.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:38.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj279 testdata.528094 2024-09-13T15:56:38.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:38.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj280 testdata.528094 2024-09-13T15:56:38.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:38.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj281 testdata.528094 2024-09-13T15:56:39.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:39.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj282 testdata.528094 2024-09-13T15:56:39.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:39.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj283 testdata.528094 2024-09-13T15:56:39.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:39.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj284 testdata.528094 2024-09-13T15:56:39.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:39.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj285 testdata.528094 2024-09-13T15:56:39.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:39.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj286 testdata.528094 2024-09-13T15:56:40.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:40.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj287 testdata.528094 2024-09-13T15:56:40.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:40.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj288 testdata.528094 2024-09-13T15:56:40.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:40.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj289 testdata.528094 2024-09-13T15:56:40.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:40.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj290 testdata.528094 2024-09-13T15:56:40.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:40.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj291 testdata.528094 2024-09-13T15:56:41.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:41.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj292 testdata.528094 2024-09-13T15:56:41.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:41.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj293 testdata.528094 2024-09-13T15:56:41.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:41.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj294 testdata.528094 2024-09-13T15:56:41.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:41.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj295 testdata.528094 2024-09-13T15:56:41.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:41.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj296 testdata.528094 2024-09-13T15:56:42.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:42.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj297 testdata.528094 2024-09-13T15:56:42.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:42.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj298 testdata.528094 2024-09-13T15:56:42.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:42.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj299 testdata.528094 2024-09-13T15:56:42.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:42.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj300 testdata.528094 2024-09-13T15:56:43.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:43.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj301 testdata.528094 2024-09-13T15:56:43.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:43.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj302 testdata.528094 2024-09-13T15:56:43.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:43.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj303 testdata.528094 2024-09-13T15:56:43.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:43.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj304 testdata.528094 2024-09-13T15:56:43.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:43.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj305 testdata.528094 2024-09-13T15:56:44.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:44.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj306 testdata.528094 2024-09-13T15:56:44.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:44.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj307 testdata.528094 2024-09-13T15:56:44.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:44.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj308 testdata.528094 2024-09-13T15:56:44.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:44.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj309 testdata.528094 2024-09-13T15:56:44.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:44.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj310 testdata.528094 2024-09-13T15:56:44.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:44.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj311 testdata.528094 2024-09-13T15:56:45.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:45.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj312 testdata.528094 2024-09-13T15:56:45.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:45.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj313 testdata.528094 2024-09-13T15:56:45.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:45.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj314 testdata.528094 2024-09-13T15:56:45.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:45.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj315 testdata.528094 2024-09-13T15:56:45.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:45.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj316 testdata.528094 2024-09-13T15:56:46.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:46.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj317 testdata.528094 2024-09-13T15:56:46.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:46.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj318 testdata.528094 2024-09-13T15:56:46.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:46.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj319 testdata.528094 2024-09-13T15:56:46.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:46.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj320 testdata.528094 2024-09-13T15:56:47.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:47.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj321 testdata.528094 2024-09-13T15:56:47.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:47.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj322 testdata.528094 2024-09-13T15:56:47.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:47.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj323 testdata.528094 2024-09-13T15:56:47.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:47.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj324 testdata.528094 2024-09-13T15:56:47.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:47.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj325 testdata.528094 2024-09-13T15:56:47.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:47.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj326 testdata.528094 2024-09-13T15:56:48.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:48.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj327 testdata.528094 2024-09-13T15:56:48.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:48.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj328 testdata.528094 2024-09-13T15:56:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:48.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj329 testdata.528094 2024-09-13T15:56:48.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:48.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj330 testdata.528094 2024-09-13T15:56:49.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:49.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj331 testdata.528094 2024-09-13T15:56:49.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:49.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj332 testdata.528094 2024-09-13T15:56:49.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:49.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj333 testdata.528094 2024-09-13T15:56:49.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:49.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj334 testdata.528094 2024-09-13T15:56:49.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:49.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj335 testdata.528094 2024-09-13T15:56:50.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:50.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj336 testdata.528094 2024-09-13T15:56:50.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:50.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj337 testdata.528094 2024-09-13T15:56:50.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:50.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj338 testdata.528094 2024-09-13T15:56:50.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:50.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj339 testdata.528094 2024-09-13T15:56:50.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:50.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj340 testdata.528094 2024-09-13T15:56:51.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:51.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj341 testdata.528094 2024-09-13T15:56:51.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:51.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj342 testdata.528094 2024-09-13T15:56:51.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:51.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj343 testdata.528094 2024-09-13T15:56:51.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:51.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj344 testdata.528094 2024-09-13T15:56:51.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:51.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj345 testdata.528094 2024-09-13T15:56:52.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:52.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj346 testdata.528094 2024-09-13T15:56:52.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:52.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj347 testdata.528094 2024-09-13T15:56:52.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:52.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj348 testdata.528094 2024-09-13T15:56:52.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:52.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj349 testdata.528094 2024-09-13T15:56:52.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:52.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj350 testdata.528094 2024-09-13T15:56:53.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:53.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj351 testdata.528094 2024-09-13T15:56:53.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:53.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj352 testdata.528094 2024-09-13T15:56:53.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:53.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj353 testdata.528094 2024-09-13T15:56:53.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:53.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj354 testdata.528094 2024-09-13T15:56:53.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:53.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj355 testdata.528094 2024-09-13T15:56:54.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:54.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj356 testdata.528094 2024-09-13T15:56:54.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:54.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj357 testdata.528094 2024-09-13T15:56:54.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:54.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj358 testdata.528094 2024-09-13T15:56:54.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:54.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj359 testdata.528094 2024-09-13T15:56:54.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:54.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj360 testdata.528094 2024-09-13T15:56:55.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:55.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj361 testdata.528094 2024-09-13T15:56:55.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:55.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj362 testdata.528094 2024-09-13T15:56:55.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:55.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj363 testdata.528094 2024-09-13T15:56:55.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:55.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj364 testdata.528094 2024-09-13T15:56:55.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:55.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj365 testdata.528094 2024-09-13T15:56:56.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:56.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj366 testdata.528094 2024-09-13T15:56:56.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:56.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj367 testdata.528094 2024-09-13T15:56:56.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:56.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj368 testdata.528094 2024-09-13T15:56:56.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:56.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj369 testdata.528094 2024-09-13T15:56:56.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:56.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj370 testdata.528094 2024-09-13T15:56:57.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:57.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj371 testdata.528094 2024-09-13T15:56:57.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:57.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj372 testdata.528094 2024-09-13T15:56:57.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:57.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj373 testdata.528094 2024-09-13T15:56:57.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:57.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj374 testdata.528094 2024-09-13T15:56:57.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:57.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj375 testdata.528094 2024-09-13T15:56:58.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:58.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj376 testdata.528094 2024-09-13T15:56:58.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:58.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj377 testdata.528094 2024-09-13T15:56:58.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:58.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj378 testdata.528094 2024-09-13T15:56:58.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:58.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj379 testdata.528094 2024-09-13T15:56:59.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:59.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj380 testdata.528094 2024-09-13T15:56:59.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:59.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj381 testdata.528094 2024-09-13T15:56:59.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:59.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj382 testdata.528094 2024-09-13T15:56:59.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:59.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj383 testdata.528094 2024-09-13T15:56:59.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:56:59.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj384 testdata.528094 2024-09-13T15:57:00.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:00.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj385 testdata.528094 2024-09-13T15:57:00.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:00.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj386 testdata.528094 2024-09-13T15:57:00.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:00.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj387 testdata.528094 2024-09-13T15:57:00.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:00.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj388 testdata.528094 2024-09-13T15:57:00.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:00.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj389 testdata.528094 2024-09-13T15:57:01.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:01.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj390 testdata.528094 2024-09-13T15:57:01.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:01.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj391 testdata.528094 2024-09-13T15:57:01.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:01.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj392 testdata.528094 2024-09-13T15:57:01.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:01.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj393 testdata.528094 2024-09-13T15:57:01.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:01.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj394 testdata.528094 2024-09-13T15:57:02.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:02.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj395 testdata.528094 2024-09-13T15:57:02.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:02.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj396 testdata.528094 2024-09-13T15:57:02.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:02.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj397 testdata.528094 2024-09-13T15:57:02.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:02.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj398 testdata.528094 2024-09-13T15:57:02.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:02.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj399 testdata.528094 2024-09-13T15:57:03.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:03.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj400 testdata.528094 2024-09-13T15:57:03.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:03.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj401 testdata.528094 2024-09-13T15:57:03.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:03.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj402 testdata.528094 2024-09-13T15:57:03.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:03.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj403 testdata.528094 2024-09-13T15:57:03.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:03.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj404 testdata.528094 2024-09-13T15:57:04.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:04.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj405 testdata.528094 2024-09-13T15:57:04.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:04.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj406 testdata.528094 2024-09-13T15:57:04.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:04.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj407 testdata.528094 2024-09-13T15:57:04.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:04.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj408 testdata.528094 2024-09-13T15:57:04.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:04.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj409 testdata.528094 2024-09-13T15:57:05.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:05.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj410 testdata.528094 2024-09-13T15:57:05.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:05.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj411 testdata.528094 2024-09-13T15:57:05.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:05.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj412 testdata.528094 2024-09-13T15:57:05.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:05.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj413 testdata.528094 2024-09-13T15:57:06.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:06.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj414 testdata.528094 2024-09-13T15:57:06.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:06.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj415 testdata.528094 2024-09-13T15:57:06.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:06.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj416 testdata.528094 2024-09-13T15:57:07.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:07.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj417 testdata.528094 2024-09-13T15:57:07.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:07.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj418 testdata.528094 2024-09-13T15:57:07.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:07.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj419 testdata.528094 2024-09-13T15:57:07.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:07.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj420 testdata.528094 2024-09-13T15:57:07.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:07.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj421 testdata.528094 2024-09-13T15:57:07.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:07.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj422 testdata.528094 2024-09-13T15:57:08.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:08.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj423 testdata.528094 2024-09-13T15:57:08.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:08.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj424 testdata.528094 2024-09-13T15:57:08.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:08.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj425 testdata.528094 2024-09-13T15:57:08.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:08.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj426 testdata.528094 2024-09-13T15:57:09.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:09.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj427 testdata.528094 2024-09-13T15:57:09.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:09.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj428 testdata.528094 2024-09-13T15:57:09.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:09.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj429 testdata.528094 2024-09-13T15:57:09.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:09.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj430 testdata.528094 2024-09-13T15:57:09.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:09.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj431 testdata.528094 2024-09-13T15:57:10.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:10.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj432 testdata.528094 2024-09-13T15:57:10.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:10.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj433 testdata.528094 2024-09-13T15:57:10.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:10.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj434 testdata.528094 2024-09-13T15:57:10.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:10.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj435 testdata.528094 2024-09-13T15:57:10.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:10.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj436 testdata.528094 2024-09-13T15:57:11.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:11.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj437 testdata.528094 2024-09-13T15:57:11.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:11.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj438 testdata.528094 2024-09-13T15:57:11.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:11.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj439 testdata.528094 2024-09-13T15:57:11.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:11.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj440 testdata.528094 2024-09-13T15:57:11.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:11.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj441 testdata.528094 2024-09-13T15:57:12.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:12.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj442 testdata.528094 2024-09-13T15:57:12.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:12.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj443 testdata.528094 2024-09-13T15:57:12.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:12.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj444 testdata.528094 2024-09-13T15:57:12.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:12.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj445 testdata.528094 2024-09-13T15:57:12.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:12.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj446 testdata.528094 2024-09-13T15:57:13.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:13.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj447 testdata.528094 2024-09-13T15:57:13.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:13.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj448 testdata.528094 2024-09-13T15:57:13.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:13.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj449 testdata.528094 2024-09-13T15:57:13.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:13.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj450 testdata.528094 2024-09-13T15:57:13.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:13.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj451 testdata.528094 2024-09-13T15:57:14.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:14.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj452 testdata.528094 2024-09-13T15:57:14.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:14.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj453 testdata.528094 2024-09-13T15:57:14.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:14.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj454 testdata.528094 2024-09-13T15:57:14.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:14.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj455 testdata.528094 2024-09-13T15:57:14.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:14.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj456 testdata.528094 2024-09-13T15:57:15.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:15.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj457 testdata.528094 2024-09-13T15:57:15.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:15.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj458 testdata.528094 2024-09-13T15:57:15.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:15.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj459 testdata.528094 2024-09-13T15:57:15.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:15.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj460 testdata.528094 2024-09-13T15:57:15.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:15.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj461 testdata.528094 2024-09-13T15:57:16.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:16.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj462 testdata.528094 2024-09-13T15:57:16.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:16.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj463 testdata.528094 2024-09-13T15:57:16.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:16.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj464 testdata.528094 2024-09-13T15:57:16.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:16.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj465 testdata.528094 2024-09-13T15:57:17.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:17.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj466 testdata.528094 2024-09-13T15:57:17.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:17.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj467 testdata.528094 2024-09-13T15:57:17.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:17.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj468 testdata.528094 2024-09-13T15:57:17.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:17.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj469 testdata.528094 2024-09-13T15:57:17.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:17.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj470 testdata.528094 2024-09-13T15:57:18.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:18.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj471 testdata.528094 2024-09-13T15:57:18.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:18.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj472 testdata.528094 2024-09-13T15:57:18.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:18.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj473 testdata.528094 2024-09-13T15:57:18.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:18.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj474 testdata.528094 2024-09-13T15:57:18.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:18.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj475 testdata.528094 2024-09-13T15:57:19.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:19.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj476 testdata.528094 2024-09-13T15:57:19.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:19.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj477 testdata.528094 2024-09-13T15:57:19.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:19.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj478 testdata.528094 2024-09-13T15:57:19.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:19.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj479 testdata.528094 2024-09-13T15:57:19.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:19.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj480 testdata.528094 2024-09-13T15:57:20.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:20.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj481 testdata.528094 2024-09-13T15:57:20.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:20.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj482 testdata.528094 2024-09-13T15:57:20.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:20.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj483 testdata.528094 2024-09-13T15:57:20.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:20.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj484 testdata.528094 2024-09-13T15:57:20.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:20.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj485 testdata.528094 2024-09-13T15:57:21.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:21.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj486 testdata.528094 2024-09-13T15:57:21.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:21.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj487 testdata.528094 2024-09-13T15:57:21.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:21.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj488 testdata.528094 2024-09-13T15:57:21.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:21.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj489 testdata.528094 2024-09-13T15:57:21.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:21.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj490 testdata.528094 2024-09-13T15:57:22.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:22.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj491 testdata.528094 2024-09-13T15:57:22.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:22.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj492 testdata.528094 2024-09-13T15:57:22.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:22.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj493 testdata.528094 2024-09-13T15:57:22.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:22.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj494 testdata.528094 2024-09-13T15:57:22.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:22.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj495 testdata.528094 2024-09-13T15:57:23.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:23.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj496 testdata.528094 2024-09-13T15:57:23.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:23.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj497 testdata.528094 2024-09-13T15:57:23.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:23.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj498 testdata.528094 2024-09-13T15:57:23.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:23.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj499 testdata.528094 2024-09-13T15:57:23.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:23.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj500 testdata.528094 2024-09-13T15:57:24.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:24.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj501 testdata.528094 2024-09-13T15:57:24.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:24.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj502 testdata.528094 2024-09-13T15:57:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:24.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj503 testdata.528094 2024-09-13T15:57:24.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:24.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj504 testdata.528094 2024-09-13T15:57:24.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:24.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj505 testdata.528094 2024-09-13T15:57:25.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:25.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj506 testdata.528094 2024-09-13T15:57:25.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:25.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj507 testdata.528094 2024-09-13T15:57:25.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:25.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj508 testdata.528094 2024-09-13T15:57:25.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:25.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj509 testdata.528094 2024-09-13T15:57:25.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:25.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj510 testdata.528094 2024-09-13T15:57:26.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:26.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj511 testdata.528094 2024-09-13T15:57:26.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:26.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj512 testdata.528094 2024-09-13T15:57:26.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:26.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj513 testdata.528094 2024-09-13T15:57:26.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:26.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj514 testdata.528094 2024-09-13T15:57:27.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:27.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj515 testdata.528094 2024-09-13T15:57:27.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:27.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj516 testdata.528094 2024-09-13T15:57:27.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:27.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj517 testdata.528094 2024-09-13T15:57:27.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:27.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj518 testdata.528094 2024-09-13T15:57:27.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:27.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj519 testdata.528094 2024-09-13T15:57:28.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:28.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj520 testdata.528094 2024-09-13T15:57:28.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:28.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj521 testdata.528094 2024-09-13T15:57:28.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:28.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj522 testdata.528094 2024-09-13T15:57:28.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:28.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj523 testdata.528094 2024-09-13T15:57:28.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:28.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj524 testdata.528094 2024-09-13T15:57:29.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:29.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj525 testdata.528094 2024-09-13T15:57:29.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:29.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj526 testdata.528094 2024-09-13T15:57:29.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:29.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj527 testdata.528094 2024-09-13T15:57:29.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:29.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj528 testdata.528094 2024-09-13T15:57:29.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:29.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj529 testdata.528094 2024-09-13T15:57:30.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:30.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj530 testdata.528094 2024-09-13T15:57:30.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:30.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj531 testdata.528094 2024-09-13T15:57:30.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:30.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj532 testdata.528094 2024-09-13T15:57:30.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:30.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj533 testdata.528094 2024-09-13T15:57:30.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:30.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj534 testdata.528094 2024-09-13T15:57:31.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:31.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj535 testdata.528094 2024-09-13T15:57:31.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:31.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj536 testdata.528094 2024-09-13T15:57:31.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:31.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj537 testdata.528094 2024-09-13T15:57:31.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:31.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj538 testdata.528094 2024-09-13T15:57:31.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:31.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj539 testdata.528094 2024-09-13T15:57:32.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:32.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj540 testdata.528094 2024-09-13T15:57:32.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:32.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj541 testdata.528094 2024-09-13T15:57:32.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:32.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj542 testdata.528094 2024-09-13T15:57:32.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:32.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj543 testdata.528094 2024-09-13T15:57:33.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:33.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj544 testdata.528094 2024-09-13T15:57:33.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:33.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj545 testdata.528094 2024-09-13T15:57:33.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:33.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj546 testdata.528094 2024-09-13T15:57:33.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:33.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj547 testdata.528094 2024-09-13T15:57:33.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:33.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj548 testdata.528094 2024-09-13T15:57:34.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:34.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj549 testdata.528094 2024-09-13T15:57:34.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:34.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj550 testdata.528094 2024-09-13T15:57:34.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:34.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj551 testdata.528094 2024-09-13T15:57:34.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:34.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj552 testdata.528094 2024-09-13T15:57:34.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:34.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj553 testdata.528094 2024-09-13T15:57:35.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:35.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj554 testdata.528094 2024-09-13T15:57:35.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:35.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj555 testdata.528094 2024-09-13T15:57:35.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:35.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj556 testdata.528094 2024-09-13T15:57:35.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:35.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj557 testdata.528094 2024-09-13T15:57:35.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:35.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj558 testdata.528094 2024-09-13T15:57:36.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:36.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj559 testdata.528094 2024-09-13T15:57:36.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:36.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj560 testdata.528094 2024-09-13T15:57:36.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:36.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj561 testdata.528094 2024-09-13T15:57:36.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:36.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj562 testdata.528094 2024-09-13T15:57:37.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:37.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj563 testdata.528094 2024-09-13T15:57:37.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:37.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj564 testdata.528094 2024-09-13T15:57:37.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:37.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj565 testdata.528094 2024-09-13T15:57:37.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:37.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj566 testdata.528094 2024-09-13T15:57:38.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:38.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj567 testdata.528094 2024-09-13T15:57:38.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:38.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj568 testdata.528094 2024-09-13T15:57:38.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:38.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj569 testdata.528094 2024-09-13T15:57:38.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:38.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj570 testdata.528094 2024-09-13T15:57:39.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:39.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj571 testdata.528094 2024-09-13T15:57:39.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:39.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj572 testdata.528094 2024-09-13T15:57:39.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:39.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj573 testdata.528094 2024-09-13T15:57:39.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:39.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj574 testdata.528094 2024-09-13T15:57:39.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:39.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj575 testdata.528094 2024-09-13T15:57:39.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:39.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj576 testdata.528094 2024-09-13T15:57:40.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:40.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj577 testdata.528094 2024-09-13T15:57:40.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:40.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj578 testdata.528094 2024-09-13T15:57:40.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:40.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj579 testdata.528094 2024-09-13T15:57:40.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:40.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj580 testdata.528094 2024-09-13T15:57:40.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:40.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj581 testdata.528094 2024-09-13T15:57:41.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:41.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj582 testdata.528094 2024-09-13T15:57:41.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:41.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj583 testdata.528094 2024-09-13T15:57:41.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:41.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj584 testdata.528094 2024-09-13T15:57:41.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:41.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj585 testdata.528094 2024-09-13T15:57:42.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:42.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj586 testdata.528094 2024-09-13T15:57:42.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:42.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj587 testdata.528094 2024-09-13T15:57:42.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:42.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj588 testdata.528094 2024-09-13T15:57:42.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:42.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj589 testdata.528094 2024-09-13T15:57:42.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:42.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj590 testdata.528094 2024-09-13T15:57:43.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:43.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj591 testdata.528094 2024-09-13T15:57:43.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:43.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj592 testdata.528094 2024-09-13T15:57:43.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:43.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj593 testdata.528094 2024-09-13T15:57:43.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:43.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj594 testdata.528094 2024-09-13T15:57:43.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:43.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj595 testdata.528094 2024-09-13T15:57:44.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:44.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj596 testdata.528094 2024-09-13T15:57:44.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:44.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj597 testdata.528094 2024-09-13T15:57:44.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:44.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj598 testdata.528094 2024-09-13T15:57:44.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:44.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj599 testdata.528094 2024-09-13T15:57:44.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:44.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj600 testdata.528094 2024-09-13T15:57:45.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:45.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj601 testdata.528094 2024-09-13T15:57:45.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:45.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj602 testdata.528094 2024-09-13T15:57:45.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:45.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj603 testdata.528094 2024-09-13T15:57:45.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:45.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj604 testdata.528094 2024-09-13T15:57:45.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:45.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj605 testdata.528094 2024-09-13T15:57:46.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:46.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj606 testdata.528094 2024-09-13T15:57:46.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:46.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj607 testdata.528094 2024-09-13T15:57:46.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:46.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj608 testdata.528094 2024-09-13T15:57:46.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:46.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj609 testdata.528094 2024-09-13T15:57:46.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:46.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj610 testdata.528094 2024-09-13T15:57:47.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:47.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj611 testdata.528094 2024-09-13T15:57:47.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:47.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj612 testdata.528094 2024-09-13T15:57:47.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:47.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj613 testdata.528094 2024-09-13T15:57:47.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:47.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj614 testdata.528094 2024-09-13T15:57:47.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:47.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj615 testdata.528094 2024-09-13T15:57:48.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:48.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj616 testdata.528094 2024-09-13T15:57:48.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:48.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj617 testdata.528094 2024-09-13T15:57:48.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:48.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj618 testdata.528094 2024-09-13T15:57:48.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:48.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj619 testdata.528094 2024-09-13T15:57:48.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:48.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj620 testdata.528094 2024-09-13T15:57:49.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:49.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj621 testdata.528094 2024-09-13T15:57:49.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:49.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj622 testdata.528094 2024-09-13T15:57:49.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:49.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj623 testdata.528094 2024-09-13T15:57:49.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:49.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj624 testdata.528094 2024-09-13T15:57:49.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:49.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj625 testdata.528094 2024-09-13T15:57:50.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:50.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj626 testdata.528094 2024-09-13T15:57:50.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:50.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj627 testdata.528094 2024-09-13T15:57:50.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:50.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj628 testdata.528094 2024-09-13T15:57:50.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:50.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj629 testdata.528094 2024-09-13T15:57:50.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:50.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj630 testdata.528094 2024-09-13T15:57:51.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:51.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj631 testdata.528094 2024-09-13T15:57:51.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:51.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj632 testdata.528094 2024-09-13T15:57:51.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:51.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj633 testdata.528094 2024-09-13T15:57:51.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:51.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj634 testdata.528094 2024-09-13T15:57:51.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:51.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj635 testdata.528094 2024-09-13T15:57:52.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:52.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj636 testdata.528094 2024-09-13T15:57:52.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:52.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj637 testdata.528094 2024-09-13T15:57:52.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:52.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj638 testdata.528094 2024-09-13T15:57:52.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:52.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj639 testdata.528094 2024-09-13T15:57:52.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:52.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj640 testdata.528094 2024-09-13T15:57:52.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:52.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj641 testdata.528094 2024-09-13T15:57:53.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:53.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj642 testdata.528094 2024-09-13T15:57:53.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:53.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj643 testdata.528094 2024-09-13T15:57:53.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:53.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj644 testdata.528094 2024-09-13T15:57:53.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:53.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj645 testdata.528094 2024-09-13T15:57:53.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:53.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj646 testdata.528094 2024-09-13T15:57:54.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:54.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj647 testdata.528094 2024-09-13T15:57:54.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:54.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj648 testdata.528094 2024-09-13T15:57:54.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:54.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj649 testdata.528094 2024-09-13T15:57:54.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:54.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj650 testdata.528094 2024-09-13T15:57:54.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:54.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj651 testdata.528094 2024-09-13T15:57:55.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:55.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj652 testdata.528094 2024-09-13T15:57:55.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:55.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj653 testdata.528094 2024-09-13T15:57:55.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:55.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj654 testdata.528094 2024-09-13T15:57:55.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:55.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj655 testdata.528094 2024-09-13T15:57:55.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:55.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj656 testdata.528094 2024-09-13T15:57:56.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:56.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj657 testdata.528094 2024-09-13T15:57:56.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:56.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj658 testdata.528094 2024-09-13T15:57:56.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:56.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj659 testdata.528094 2024-09-13T15:57:56.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:56.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj660 testdata.528094 2024-09-13T15:57:57.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:57.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj661 testdata.528094 2024-09-13T15:57:57.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:57.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj662 testdata.528094 2024-09-13T15:57:57.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:57.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj663 testdata.528094 2024-09-13T15:57:57.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:57.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj664 testdata.528094 2024-09-13T15:57:57.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:57.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj665 testdata.528094 2024-09-13T15:57:58.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:58.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj666 testdata.528094 2024-09-13T15:57:58.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:58.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj667 testdata.528094 2024-09-13T15:57:58.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:58.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj668 testdata.528094 2024-09-13T15:57:58.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:58.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj669 testdata.528094 2024-09-13T15:57:58.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:58.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj670 testdata.528094 2024-09-13T15:57:59.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:59.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj671 testdata.528094 2024-09-13T15:57:59.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:59.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj672 testdata.528094 2024-09-13T15:57:59.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:59.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj673 testdata.528094 2024-09-13T15:57:59.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:59.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj674 testdata.528094 2024-09-13T15:57:59.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:57:59.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj675 testdata.528094 2024-09-13T15:58:00.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:00.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj676 testdata.528094 2024-09-13T15:58:00.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:00.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj677 testdata.528094 2024-09-13T15:58:00.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:00.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj678 testdata.528094 2024-09-13T15:58:00.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:00.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj679 testdata.528094 2024-09-13T15:58:00.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:00.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj680 testdata.528094 2024-09-13T15:58:01.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:01.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj681 testdata.528094 2024-09-13T15:58:01.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:01.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj682 testdata.528094 2024-09-13T15:58:01.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:01.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj683 testdata.528094 2024-09-13T15:58:01.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:01.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj684 testdata.528094 2024-09-13T15:58:01.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:01.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj685 testdata.528094 2024-09-13T15:58:02.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:02.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj686 testdata.528094 2024-09-13T15:58:02.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:02.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj687 testdata.528094 2024-09-13T15:58:02.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:02.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj688 testdata.528094 2024-09-13T15:58:02.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:02.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj689 testdata.528094 2024-09-13T15:58:02.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:02.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj690 testdata.528094 2024-09-13T15:58:03.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:03.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj691 testdata.528094 2024-09-13T15:58:03.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:03.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj692 testdata.528094 2024-09-13T15:58:03.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:03.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj693 testdata.528094 2024-09-13T15:58:03.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:03.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj694 testdata.528094 2024-09-13T15:58:03.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:03.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj695 testdata.528094 2024-09-13T15:58:04.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:04.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj696 testdata.528094 2024-09-13T15:58:04.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:04.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj697 testdata.528094 2024-09-13T15:58:04.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:04.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj698 testdata.528094 2024-09-13T15:58:04.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:04.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj699 testdata.528094 2024-09-13T15:58:04.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:04.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj700 testdata.528094 2024-09-13T15:58:05.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:05.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj701 testdata.528094 2024-09-13T15:58:05.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:05.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj702 testdata.528094 2024-09-13T15:58:05.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:05.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj703 testdata.528094 2024-09-13T15:58:05.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:05.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj704 testdata.528094 2024-09-13T15:58:05.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:05.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj705 testdata.528094 2024-09-13T15:58:06.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:06.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj706 testdata.528094 2024-09-13T15:58:06.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:06.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj707 testdata.528094 2024-09-13T15:58:06.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:06.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj708 testdata.528094 2024-09-13T15:58:06.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:06.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj709 testdata.528094 2024-09-13T15:58:06.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:06.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj710 testdata.528094 2024-09-13T15:58:07.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:07.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj711 testdata.528094 2024-09-13T15:58:07.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:07.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj712 testdata.528094 2024-09-13T15:58:07.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:07.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj713 testdata.528094 2024-09-13T15:58:07.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:07.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj714 testdata.528094 2024-09-13T15:58:07.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:07.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj715 testdata.528094 2024-09-13T15:58:08.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:08.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj716 testdata.528094 2024-09-13T15:58:08.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:08.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj717 testdata.528094 2024-09-13T15:58:09.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:09.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj718 testdata.528094 2024-09-13T15:58:09.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:09.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj719 testdata.528094 2024-09-13T15:58:09.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:09.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj720 testdata.528094 2024-09-13T15:58:09.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:09.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj721 testdata.528094 2024-09-13T15:58:09.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:09.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj722 testdata.528094 2024-09-13T15:58:10.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:10.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj723 testdata.528094 2024-09-13T15:58:10.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:10.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj724 testdata.528094 2024-09-13T15:58:10.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:10.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj725 testdata.528094 2024-09-13T15:58:10.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:10.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj726 testdata.528094 2024-09-13T15:58:10.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:10.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj727 testdata.528094 2024-09-13T15:58:11.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:11.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj728 testdata.528094 2024-09-13T15:58:11.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:11.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj729 testdata.528094 2024-09-13T15:58:11.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:11.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj730 testdata.528094 2024-09-13T15:58:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:11.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj731 testdata.528094 2024-09-13T15:58:11.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:11.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj732 testdata.528094 2024-09-13T15:58:12.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:12.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj733 testdata.528094 2024-09-13T15:58:12.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:12.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj734 testdata.528094 2024-09-13T15:58:12.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:12.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj735 testdata.528094 2024-09-13T15:58:12.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:12.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj736 testdata.528094 2024-09-13T15:58:12.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:12.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj737 testdata.528094 2024-09-13T15:58:13.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:13.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj738 testdata.528094 2024-09-13T15:58:13.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:13.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj739 testdata.528094 2024-09-13T15:58:13.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:13.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj740 testdata.528094 2024-09-13T15:58:13.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:13.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj741 testdata.528094 2024-09-13T15:58:13.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:13.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj742 testdata.528094 2024-09-13T15:58:14.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:14.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj743 testdata.528094 2024-09-13T15:58:14.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:14.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj744 testdata.528094 2024-09-13T15:58:14.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:14.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj745 testdata.528094 2024-09-13T15:58:14.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:14.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj746 testdata.528094 2024-09-13T15:58:14.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:14.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj747 testdata.528094 2024-09-13T15:58:15.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:15.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj748 testdata.528094 2024-09-13T15:58:15.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:15.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj749 testdata.528094 2024-09-13T15:58:15.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:15.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj750 testdata.528094 2024-09-13T15:58:15.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:15.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj751 testdata.528094 2024-09-13T15:58:15.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:15.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj752 testdata.528094 2024-09-13T15:58:16.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:16.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj753 testdata.528094 2024-09-13T15:58:16.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:16.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj754 testdata.528094 2024-09-13T15:58:16.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:16.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj755 testdata.528094 2024-09-13T15:58:16.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:16.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj756 testdata.528094 2024-09-13T15:58:16.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:16.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj757 testdata.528094 2024-09-13T15:58:17.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:17.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj758 testdata.528094 2024-09-13T15:58:17.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:17.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj759 testdata.528094 2024-09-13T15:58:17.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:17.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj760 testdata.528094 2024-09-13T15:58:17.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:17.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj761 testdata.528094 2024-09-13T15:58:17.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:17.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj762 testdata.528094 2024-09-13T15:58:18.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:18.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj763 testdata.528094 2024-09-13T15:58:18.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:18.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj764 testdata.528094 2024-09-13T15:58:18.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:18.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj765 testdata.528094 2024-09-13T15:58:18.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:18.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj766 testdata.528094 2024-09-13T15:58:18.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:18.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj767 testdata.528094 2024-09-13T15:58:19.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:19.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj768 testdata.528094 2024-09-13T15:58:19.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:19.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj769 testdata.528094 2024-09-13T15:58:19.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:19.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj770 testdata.528094 2024-09-13T15:58:19.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:19.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj771 testdata.528094 2024-09-13T15:58:19.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:19.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj772 testdata.528094 2024-09-13T15:58:20.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:20.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj773 testdata.528094 2024-09-13T15:58:20.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:20.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj774 testdata.528094 2024-09-13T15:58:20.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:20.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj775 testdata.528094 2024-09-13T15:58:20.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:20.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj776 testdata.528094 2024-09-13T15:58:20.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:20.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj777 testdata.528094 2024-09-13T15:58:21.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:21.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj778 testdata.528094 2024-09-13T15:58:21.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:21.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj779 testdata.528094 2024-09-13T15:58:21.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:21.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj780 testdata.528094 2024-09-13T15:58:21.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:21.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj781 testdata.528094 2024-09-13T15:58:21.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:21.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj782 testdata.528094 2024-09-13T15:58:22.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:22.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj783 testdata.528094 2024-09-13T15:58:22.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:22.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj784 testdata.528094 2024-09-13T15:58:22.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:22.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj785 testdata.528094 2024-09-13T15:58:22.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:22.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj786 testdata.528094 2024-09-13T15:58:22.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:22.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj787 testdata.528094 2024-09-13T15:58:23.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:23.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj788 testdata.528094 2024-09-13T15:58:23.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:23.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj789 testdata.528094 2024-09-13T15:58:23.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:23.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj790 testdata.528094 2024-09-13T15:58:23.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:23.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj791 testdata.528094 2024-09-13T15:58:23.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:23.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj792 testdata.528094 2024-09-13T15:58:24.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:24.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj793 testdata.528094 2024-09-13T15:58:24.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:24.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj794 testdata.528094 2024-09-13T15:58:24.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:24.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj795 testdata.528094 2024-09-13T15:58:24.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:24.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj796 testdata.528094 2024-09-13T15:58:25.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:25.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj797 testdata.528094 2024-09-13T15:58:25.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:25.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj798 testdata.528094 2024-09-13T15:58:25.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:25.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj799 testdata.528094 2024-09-13T15:58:25.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:25.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj800 testdata.528094 2024-09-13T15:58:25.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:25.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj801 testdata.528094 2024-09-13T15:58:26.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:26.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj802 testdata.528094 2024-09-13T15:58:26.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:26.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj803 testdata.528094 2024-09-13T15:58:26.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:26.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj804 testdata.528094 2024-09-13T15:58:26.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:26.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj805 testdata.528094 2024-09-13T15:58:26.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:26.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj806 testdata.528094 2024-09-13T15:58:27.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:27.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj807 testdata.528094 2024-09-13T15:58:27.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:27.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj808 testdata.528094 2024-09-13T15:58:27.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:27.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj809 testdata.528094 2024-09-13T15:58:27.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:27.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj810 testdata.528094 2024-09-13T15:58:27.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:27.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj811 testdata.528094 2024-09-13T15:58:27.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:27.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj812 testdata.528094 2024-09-13T15:58:28.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:28.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj813 testdata.528094 2024-09-13T15:58:28.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:28.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj814 testdata.528094 2024-09-13T15:58:28.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:28.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj815 testdata.528094 2024-09-13T15:58:28.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:28.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj816 testdata.528094 2024-09-13T15:58:29.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:29.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj817 testdata.528094 2024-09-13T15:58:29.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:29.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj818 testdata.528094 2024-09-13T15:58:29.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:29.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj819 testdata.528094 2024-09-13T15:58:29.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:29.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj820 testdata.528094 2024-09-13T15:58:29.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:29.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj821 testdata.528094 2024-09-13T15:58:30.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:30.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj822 testdata.528094 2024-09-13T15:58:30.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:30.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj823 testdata.528094 2024-09-13T15:58:30.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:30.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj824 testdata.528094 2024-09-13T15:58:30.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:30.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj825 testdata.528094 2024-09-13T15:58:30.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:30.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj826 testdata.528094 2024-09-13T15:58:31.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:31.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj827 testdata.528094 2024-09-13T15:58:31.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:31.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj828 testdata.528094 2024-09-13T15:58:31.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:31.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj829 testdata.528094 2024-09-13T15:58:31.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:31.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj830 testdata.528094 2024-09-13T15:58:31.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:31.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj831 testdata.528094 2024-09-13T15:58:31.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:31.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj832 testdata.528094 2024-09-13T15:58:32.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:32.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj833 testdata.528094 2024-09-13T15:58:32.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:32.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj834 testdata.528094 2024-09-13T15:58:32.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:32.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj835 testdata.528094 2024-09-13T15:58:32.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:32.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj836 testdata.528094 2024-09-13T15:58:32.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:32.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj837 testdata.528094 2024-09-13T15:58:33.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:33.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj838 testdata.528094 2024-09-13T15:58:33.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:33.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj839 testdata.528094 2024-09-13T15:58:33.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:33.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj840 testdata.528094 2024-09-13T15:58:33.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:33.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj841 testdata.528094 2024-09-13T15:58:33.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:33.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj842 testdata.528094 2024-09-13T15:58:34.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:34.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj843 testdata.528094 2024-09-13T15:58:34.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:34.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj844 testdata.528094 2024-09-13T15:58:34.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:34.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj845 testdata.528094 2024-09-13T15:58:34.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:34.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj846 testdata.528094 2024-09-13T15:58:35.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:35.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj847 testdata.528094 2024-09-13T15:58:35.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:35.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj848 testdata.528094 2024-09-13T15:58:35.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:35.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj849 testdata.528094 2024-09-13T15:58:35.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:35.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj850 testdata.528094 2024-09-13T15:58:35.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:35.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj851 testdata.528094 2024-09-13T15:58:36.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:36.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj852 testdata.528094 2024-09-13T15:58:36.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:36.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj853 testdata.528094 2024-09-13T15:58:36.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:36.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj854 testdata.528094 2024-09-13T15:58:36.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:36.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj855 testdata.528094 2024-09-13T15:58:36.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:36.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj856 testdata.528094 2024-09-13T15:58:37.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:37.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj857 testdata.528094 2024-09-13T15:58:37.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:37.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj858 testdata.528094 2024-09-13T15:58:37.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:37.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj859 testdata.528094 2024-09-13T15:58:37.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:37.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj860 testdata.528094 2024-09-13T15:58:37.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:37.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj861 testdata.528094 2024-09-13T15:58:38.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:38.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj862 testdata.528094 2024-09-13T15:58:38.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:38.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj863 testdata.528094 2024-09-13T15:58:38.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:38.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj864 testdata.528094 2024-09-13T15:58:38.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:38.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj865 testdata.528094 2024-09-13T15:58:38.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:38.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj866 testdata.528094 2024-09-13T15:58:39.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:39.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj867 testdata.528094 2024-09-13T15:58:39.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:39.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj868 testdata.528094 2024-09-13T15:58:39.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:39.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj869 testdata.528094 2024-09-13T15:58:40.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:40.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj870 testdata.528094 2024-09-13T15:58:40.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:40.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj871 testdata.528094 2024-09-13T15:58:40.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:40.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj872 testdata.528094 2024-09-13T15:58:40.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:40.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj873 testdata.528094 2024-09-13T15:58:40.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:40.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj874 testdata.528094 2024-09-13T15:58:41.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:41.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj875 testdata.528094 2024-09-13T15:58:41.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:41.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj876 testdata.528094 2024-09-13T15:58:41.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:41.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj877 testdata.528094 2024-09-13T15:58:41.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:41.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj878 testdata.528094 2024-09-13T15:58:41.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:41.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj879 testdata.528094 2024-09-13T15:58:42.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:42.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj880 testdata.528094 2024-09-13T15:58:42.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:42.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj881 testdata.528094 2024-09-13T15:58:42.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:42.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj882 testdata.528094 2024-09-13T15:58:42.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:42.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj883 testdata.528094 2024-09-13T15:58:42.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:42.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj884 testdata.528094 2024-09-13T15:58:43.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:43.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj885 testdata.528094 2024-09-13T15:58:43.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:43.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj886 testdata.528094 2024-09-13T15:58:43.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:43.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj887 testdata.528094 2024-09-13T15:58:43.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:43.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj888 testdata.528094 2024-09-13T15:58:44.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:44.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj889 testdata.528094 2024-09-13T15:58:44.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:44.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj890 testdata.528094 2024-09-13T15:58:44.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:44.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj891 testdata.528094 2024-09-13T15:58:44.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:44.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj892 testdata.528094 2024-09-13T15:58:44.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:44.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj893 testdata.528094 2024-09-13T15:58:45.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:45.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj894 testdata.528094 2024-09-13T15:58:45.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:45.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj895 testdata.528094 2024-09-13T15:58:45.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:45.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj896 testdata.528094 2024-09-13T15:58:45.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:45.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj897 testdata.528094 2024-09-13T15:58:45.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:45.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj898 testdata.528094 2024-09-13T15:58:46.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:46.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj899 testdata.528094 2024-09-13T15:58:46.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:46.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj900 testdata.528094 2024-09-13T15:58:46.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:46.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj901 testdata.528094 2024-09-13T15:58:46.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:46.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj902 testdata.528094 2024-09-13T15:58:46.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:46.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj903 testdata.528094 2024-09-13T15:58:47.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:47.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj904 testdata.528094 2024-09-13T15:58:47.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:47.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj905 testdata.528094 2024-09-13T15:58:47.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:47.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj906 testdata.528094 2024-09-13T15:58:47.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:47.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj907 testdata.528094 2024-09-13T15:58:47.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:47.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj908 testdata.528094 2024-09-13T15:58:48.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:48.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj909 testdata.528094 2024-09-13T15:58:48.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:48.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj910 testdata.528094 2024-09-13T15:58:48.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:48.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj911 testdata.528094 2024-09-13T15:58:48.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:48.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj912 testdata.528094 2024-09-13T15:58:48.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:48.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj913 testdata.528094 2024-09-13T15:58:49.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:49.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj914 testdata.528094 2024-09-13T15:58:49.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:49.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj915 testdata.528094 2024-09-13T15:58:49.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:49.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj916 testdata.528094 2024-09-13T15:58:49.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:49.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj917 testdata.528094 2024-09-13T15:58:49.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:49.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj918 testdata.528094 2024-09-13T15:58:50.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:50.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj919 testdata.528094 2024-09-13T15:58:50.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:50.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj920 testdata.528094 2024-09-13T15:58:50.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:50.416 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj921 testdata.528094 2024-09-13T15:58:50.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:50.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj922 testdata.528094 2024-09-13T15:58:50.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:50.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj923 testdata.528094 2024-09-13T15:58:51.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:51.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj924 testdata.528094 2024-09-13T15:58:51.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:51.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj925 testdata.528094 2024-09-13T15:58:51.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:51.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj926 testdata.528094 2024-09-13T15:58:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj927 testdata.528094 2024-09-13T15:58:51.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:51.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj928 testdata.528094 2024-09-13T15:58:52.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:52.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj929 testdata.528094 2024-09-13T15:58:52.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:52.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj930 testdata.528094 2024-09-13T15:58:52.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:52.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj931 testdata.528094 2024-09-13T15:58:52.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:52.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj932 testdata.528094 2024-09-13T15:58:52.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:52.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj933 testdata.528094 2024-09-13T15:58:53.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:53.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj934 testdata.528094 2024-09-13T15:58:53.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:53.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj935 testdata.528094 2024-09-13T15:58:53.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:53.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj936 testdata.528094 2024-09-13T15:58:53.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:53.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj937 testdata.528094 2024-09-13T15:58:53.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:53.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj938 testdata.528094 2024-09-13T15:58:54.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:54.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj939 testdata.528094 2024-09-13T15:58:54.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:54.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj940 testdata.528094 2024-09-13T15:58:54.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:54.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj941 testdata.528094 2024-09-13T15:58:54.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:54.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj942 testdata.528094 2024-09-13T15:58:55.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:55.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj943 testdata.528094 2024-09-13T15:58:55.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:55.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj944 testdata.528094 2024-09-13T15:58:55.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:55.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj945 testdata.528094 2024-09-13T15:58:55.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:55.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj946 testdata.528094 2024-09-13T15:58:55.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:55.897 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj947 testdata.528094 2024-09-13T15:58:56.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:56.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj948 testdata.528094 2024-09-13T15:58:56.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:56.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj949 testdata.528094 2024-09-13T15:58:56.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:56.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj950 testdata.528094 2024-09-13T15:58:56.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:56.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj951 testdata.528094 2024-09-13T15:58:57.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:57.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj952 testdata.528094 2024-09-13T15:58:57.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:57.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj953 testdata.528094 2024-09-13T15:58:57.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:57.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj954 testdata.528094 2024-09-13T15:58:57.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:57.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj955 testdata.528094 2024-09-13T15:58:57.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:57.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj956 testdata.528094 2024-09-13T15:58:58.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:58.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj957 testdata.528094 2024-09-13T15:58:58.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:58.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj958 testdata.528094 2024-09-13T15:58:58.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:58.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj959 testdata.528094 2024-09-13T15:58:58.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:58.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj960 testdata.528094 2024-09-13T15:58:58.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:58.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj961 testdata.528094 2024-09-13T15:58:59.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:59.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj962 testdata.528094 2024-09-13T15:58:59.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:59.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj963 testdata.528094 2024-09-13T15:58:59.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:59.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj964 testdata.528094 2024-09-13T15:58:59.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:59.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj965 testdata.528094 2024-09-13T15:58:59.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:58:59.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj966 testdata.528094 2024-09-13T15:59:00.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:00.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj967 testdata.528094 2024-09-13T15:59:00.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:00.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj968 testdata.528094 2024-09-13T15:59:00.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:00.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj969 testdata.528094 2024-09-13T15:59:00.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:00.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj970 testdata.528094 2024-09-13T15:59:00.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:00.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj971 testdata.528094 2024-09-13T15:59:01.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:01.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj972 testdata.528094 2024-09-13T15:59:01.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:01.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj973 testdata.528094 2024-09-13T15:59:01.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:01.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj974 testdata.528094 2024-09-13T15:59:01.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:01.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj975 testdata.528094 2024-09-13T15:59:01.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:01.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj976 testdata.528094 2024-09-13T15:59:02.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:02.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj977 testdata.528094 2024-09-13T15:59:02.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:02.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj978 testdata.528094 2024-09-13T15:59:02.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:02.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj979 testdata.528094 2024-09-13T15:59:02.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:02.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj980 testdata.528094 2024-09-13T15:59:02.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:02.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj981 testdata.528094 2024-09-13T15:59:03.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:03.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj982 testdata.528094 2024-09-13T15:59:03.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:03.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj983 testdata.528094 2024-09-13T15:59:03.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:03.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj984 testdata.528094 2024-09-13T15:59:03.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:03.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj985 testdata.528094 2024-09-13T15:59:03.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:03.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj986 testdata.528094 2024-09-13T15:59:04.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:04.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj987 testdata.528094 2024-09-13T15:59:04.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:04.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj988 testdata.528094 2024-09-13T15:59:04.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:04.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj989 testdata.528094 2024-09-13T15:59:04.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:04.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj990 testdata.528094 2024-09-13T15:59:05.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:05.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj991 testdata.528094 2024-09-13T15:59:05.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:05.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj992 testdata.528094 2024-09-13T15:59:05.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:05.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj993 testdata.528094 2024-09-13T15:59:05.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:05.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj994 testdata.528094 2024-09-13T15:59:05.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:05.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj995 testdata.528094 2024-09-13T15:59:06.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:06.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj996 testdata.528094 2024-09-13T15:59:06.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:06.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj997 testdata.528094 2024-09-13T15:59:06.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:06.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj998 testdata.528094 2024-09-13T15:59:06.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:06.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj999 testdata.528094 2024-09-13T15:59:06.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T15:59:06.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj1000 testdata.528094 2024-09-13T15:59:07.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:341: _scrub_abort: rm -f testdata.528094 2024-09-13T15:59:07.038 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:343: _scrub_abort: get_primary test obj1 2024-09-13T15:59:07.039 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-13T15:59:07.039 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T15:59:07.039 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-13T15:59:07.039 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T15:59:07.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:343: _scrub_abort: local primary=1 2024-09-13T15:59:07.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: local pgid=1.0 2024-09-13T15:59:07.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:346: _scrub_abort: ceph tell 1.0 schedule-deep-scrub 2024-09-13T15:59:07.503 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T15:59:07.503 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T15:59:07.503 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T15:59:07.503 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-08-30T15:57:27.504191+0000" 2024-09-13T15:59:07.503 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T15:59:07.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: set -o pipefail 2024-09-13T15:59:07.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:350: _scrub_abort: found=no 2024-09-13T15:59:07.518 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: seq 0 200 2024-09-13T15:59:07.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: for i in $(seq 0 200) 2024-09-13T15:59:07.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:353: _scrub_abort: flush_pg_stats 2024-09-13T15:59:07.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:59:07.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:59:07.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:59:07.841 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:59:07.841 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:59:07.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:59:07.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:07.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:59:07.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836533 2024-09-13T15:59:07.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836533 2024-09-13T15:59:07.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533' 2024-09-13T15:59:07.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:07.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:59:08.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673009 2024-09-13T15:59:08.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673009 2024-09-13T15:59:08.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-42949673009' 2024-09-13T15:59:08.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:08.127 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:59:08.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509486 2024-09-13T15:59:08.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509486 2024-09-13T15:59:08.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-42949673009 2-64424509486' 2024-09-13T15:59:08.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:08.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836533 2024-09-13T15:59:08.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:08.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:59:08.269 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836533 2024-09-13T15:59:08.269 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:08.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836533 2024-09-13T15:59:08.271 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836533 2024-09-13T15:59:08.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-09-13T15:59:08.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:08.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836533 -lt 21474836533 2024-09-13T15:59:08.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:08.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673009 2024-09-13T15:59:08.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:08.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:59:08.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673009 2024-09-13T15:59:08.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:08.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673009 2024-09-13T15:59:08.606 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673009 2024-09-13T15:59:08.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673009' 2024-09-13T15:59:08.606 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:59:08.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673009 -lt 42949673009 2024-09-13T15:59:08.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:08.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:08.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509486 2024-09-13T15:59:08.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:59:08.921 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509486 2024-09-13T15:59:08.921 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:08.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509486 2024-09-13T15:59:08.923 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509486 2024-09-13T15:59:08.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509486' 2024-09-13T15:59:08.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:09.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509486 -lt 64424509486 2024-09-13T15:59:09.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: ceph pg dump pgs 2024-09-13T15:59:09.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep -q scrubbing 2024-09-13T15:59:09.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep '^1.0' 2024-09-13T15:59:09.521 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:59:09.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:356: _scrub_abort: found=yes 2024-09-13T15:59:09.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:358: _scrub_abort: break 2024-09-13T15:59:09.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: set +o pipefail 2024-09-13T15:59:09.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:363: _scrub_abort: test yes = no 2024-09-13T15:59:09.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:369: _scrub_abort: ceph osd set nodeep-scrub 2024-09-13T15:59:10.138 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T15:59:10.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2024-09-13T15:59:10.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: ceph osd set noscrub 2024-09-13T15:59:10.715 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T15:59:10.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:376: _scrub_abort: set -o pipefail 2024-09-13T15:59:10.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: seq 0 200 2024-09-13T15:59:10.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T15:59:10.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T15:59:10.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:59:10.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:59:11.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:59:11.053 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:59:11.053 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:59:11.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:59:11.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:11.053 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:59:11.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836535 2024-09-13T15:59:11.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836535 2024-09-13T15:59:11.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535' 2024-09-13T15:59:11.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:11.196 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:59:11.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673011 2024-09-13T15:59:11.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673011 2024-09-13T15:59:11.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535 1-42949673011' 2024-09-13T15:59:11.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:11.334 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:59:11.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509487 2024-09-13T15:59:11.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509487 2024-09-13T15:59:11.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535 1-42949673011 2-64424509487' 2024-09-13T15:59:11.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:11.472 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836535 2024-09-13T15:59:11.472 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:11.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:59:11.475 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836535 2024-09-13T15:59:11.475 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:11.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836535 2024-09-13T15:59:11.476 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836535 2024-09-13T15:59:11.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-09-13T15:59:11.476 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:11.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836534 -lt 21474836535 2024-09-13T15:59:11.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:59:12.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:59:12.782 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:13.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836535 -lt 21474836535 2024-09-13T15:59:13.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:13.110 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673011 2024-09-13T15:59:13.110 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:13.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:59:13.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673011 2024-09-13T15:59:13.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:13.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673011 2024-09-13T15:59:13.114 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673011 2024-09-13T15:59:13.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2024-09-13T15:59:13.114 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:59:13.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673011 -lt 42949673011 2024-09-13T15:59:13.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:13.487 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509487 2024-09-13T15:59:13.487 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:13.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:59:13.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509487 2024-09-13T15:59:13.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:13.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509487 2024-09-13T15:59:13.491 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509487 2024-09-13T15:59:13.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509487' 2024-09-13T15:59:13.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:13.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509488 -lt 64424509487 2024-09-13T15:59:13.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T15:59:13.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T15:59:13.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T15:59:14.080 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:59:14.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T15:59:14.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T15:59:14.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T15:59:14.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:59:14.096 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:59:14.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:59:14.404 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:59:14.405 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:59:14.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:59:14.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:14.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:59:14.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836536 2024-09-13T15:59:14.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836536 2024-09-13T15:59:14.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536' 2024-09-13T15:59:14.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:14.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:59:14.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673012 2024-09-13T15:59:14.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673012 2024-09-13T15:59:14.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-42949673012' 2024-09-13T15:59:14.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:14.940 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:59:15.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509489 2024-09-13T15:59:15.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509489 2024-09-13T15:59:15.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-42949673012 2-64424509489' 2024-09-13T15:59:15.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:15.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836536 2024-09-13T15:59:15.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:15.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:59:15.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836536 2024-09-13T15:59:15.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:15.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836536 2024-09-13T15:59:15.110 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836536 2024-09-13T15:59:15.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-09-13T15:59:15.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:15.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836535 -lt 21474836536 2024-09-13T15:59:15.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:59:16.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:59:16.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:16.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836537 -lt 21474836536 2024-09-13T15:59:16.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:16.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:16.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673012 2024-09-13T15:59:16.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:59:16.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673012 2024-09-13T15:59:16.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:16.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673012 2024-09-13T15:59:16.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673012' 2024-09-13T15:59:16.744 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673012 2024-09-13T15:59:16.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:59:17.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673013 -lt 42949673012 2024-09-13T15:59:17.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:17.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509489 2024-09-13T15:59:17.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:17.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:59:17.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509489 2024-09-13T15:59:17.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:17.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509489 2024-09-13T15:59:17.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509489' 2024-09-13T15:59:17.049 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509489 2024-09-13T15:59:17.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:17.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509489 -lt 64424509489 2024-09-13T15:59:17.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T15:59:17.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T15:59:17.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T15:59:17.665 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:59:17.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T15:59:17.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T15:59:17.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T15:59:17.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:59:17.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:59:17.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:59:17.987 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:59:17.987 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:59:17.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:59:17.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:17.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:59:18.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836538 2024-09-13T15:59:18.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836538 2024-09-13T15:59:18.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836538' 2024-09-13T15:59:18.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:18.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:59:18.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673014 2024-09-13T15:59:18.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673014 2024-09-13T15:59:18.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836538 1-42949673014' 2024-09-13T15:59:18.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:18.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:59:18.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509491 2024-09-13T15:59:18.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509491 2024-09-13T15:59:18.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836538 1-42949673014 2-64424509491' 2024-09-13T15:59:18.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:18.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836538 2024-09-13T15:59:18.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:18.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:59:18.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836538 2024-09-13T15:59:18.442 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:18.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836538 2024-09-13T15:59:18.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-09-13T15:59:18.443 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836538 2024-09-13T15:59:18.443 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:18.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836538 -lt 21474836538 2024-09-13T15:59:18.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:18.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:18.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673014 2024-09-13T15:59:18.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:59:18.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673014 2024-09-13T15:59:18.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:18.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673014 2024-09-13T15:59:18.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673014' 2024-09-13T15:59:18.765 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673014 2024-09-13T15:59:18.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:59:19.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673014 -lt 42949673014 2024-09-13T15:59:19.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:19.165 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509491 2024-09-13T15:59:19.165 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:19.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:59:19.167 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509491 2024-09-13T15:59:19.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:19.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509491 2024-09-13T15:59:19.169 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509491 2024-09-13T15:59:19.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509491' 2024-09-13T15:59:19.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:19.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509490 -lt 64424509491 2024-09-13T15:59:19.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:59:20.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:59:20.485 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:20.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509491 -lt 64424509491 2024-09-13T15:59:20.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T15:59:20.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T15:59:20.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T15:59:21.124 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:59:21.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T15:59:21.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T15:59:21.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T15:59:21.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:59:21.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:59:21.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:59:21.461 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:59:21.461 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:59:21.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:59:21.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:21.462 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:59:21.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836540 2024-09-13T15:59:21.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836540 2024-09-13T15:59:21.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540' 2024-09-13T15:59:21.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:21.650 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:59:21.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673016 2024-09-13T15:59:21.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673016 2024-09-13T15:59:21.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-42949673016' 2024-09-13T15:59:21.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:21.781 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:59:21.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509493 2024-09-13T15:59:21.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509493 2024-09-13T15:59:21.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-42949673016 2-64424509493' 2024-09-13T15:59:21.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:21.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836540 2024-09-13T15:59:21.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:21.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:59:21.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836540 2024-09-13T15:59:21.920 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:21.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836540 2024-09-13T15:59:21.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-09-13T15:59:21.921 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836540 2024-09-13T15:59:21.921 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:22.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836539 -lt 21474836540 2024-09-13T15:59:22.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:59:23.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:59:23.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:23.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836540 -lt 21474836540 2024-09-13T15:59:23.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:23.548 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673016 2024-09-13T15:59:23.549 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:23.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:59:23.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673016 2024-09-13T15:59:23.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:23.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673016 2024-09-13T15:59:23.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673016' 2024-09-13T15:59:23.553 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673016 2024-09-13T15:59:23.553 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:59:23.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673016 -lt 42949673016 2024-09-13T15:59:23.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:23.921 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509493 2024-09-13T15:59:23.921 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:23.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:59:23.924 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509493 2024-09-13T15:59:23.924 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:23.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509493 2024-09-13T15:59:23.925 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509493 2024-09-13T15:59:23.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509493' 2024-09-13T15:59:23.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:24.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509493 -lt 64424509493 2024-09-13T15:59:24.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T15:59:24.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T15:59:24.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T15:59:24.549 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:59:24.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T15:59:24.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T15:59:24.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T15:59:24.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:59:24.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:59:24.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:59:24.936 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:59:24.936 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:59:24.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:59:24.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:24.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:59:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836541 2024-09-13T15:59:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836541 2024-09-13T15:59:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836541' 2024-09-13T15:59:25.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:25.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:59:25.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673017 2024-09-13T15:59:25.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673017 2024-09-13T15:59:25.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836541 1-42949673017' 2024-09-13T15:59:25.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:25.198 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:59:25.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509494 2024-09-13T15:59:25.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509494 2024-09-13T15:59:25.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836541 1-42949673017 2-64424509494' 2024-09-13T15:59:25.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:25.336 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836541 2024-09-13T15:59:25.336 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:25.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:59:25.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836541 2024-09-13T15:59:25.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:25.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836541 2024-09-13T15:59:25.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-09-13T15:59:25.339 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836541 2024-09-13T15:59:25.339 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:25.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836540 -lt 21474836541 2024-09-13T15:59:25.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:59:26.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:59:26.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:27.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836542 -lt 21474836541 2024-09-13T15:59:27.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:27.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:27.026 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673017 2024-09-13T15:59:27.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:59:27.029 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673017 2024-09-13T15:59:27.029 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:27.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673017 2024-09-13T15:59:27.030 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673017 2024-09-13T15:59:27.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673017' 2024-09-13T15:59:27.030 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:59:27.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673018 -lt 42949673017 2024-09-13T15:59:27.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:27.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509494 2024-09-13T15:59:27.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:27.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:59:27.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509494 2024-09-13T15:59:27.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:27.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509494 2024-09-13T15:59:27.405 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509494 2024-09-13T15:59:27.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509494' 2024-09-13T15:59:27.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:27.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509494 -lt 64424509494 2024-09-13T15:59:27.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T15:59:27.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T15:59:27.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T15:59:28.102 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:59:28.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T15:59:28.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T15:59:28.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T15:59:28.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:59:28.119 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:59:28.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:59:28.445 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:59:28.445 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:59:28.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:59:28.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:28.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:59:28.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836543 2024-09-13T15:59:28.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836543 2024-09-13T15:59:28.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543' 2024-09-13T15:59:28.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:28.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:59:28.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673019 2024-09-13T15:59:28.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673019 2024-09-13T15:59:28.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-42949673019' 2024-09-13T15:59:28.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:28.717 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:59:28.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509496 2024-09-13T15:59:28.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509496 2024-09-13T15:59:28.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-42949673019 2-64424509496' 2024-09-13T15:59:28.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:28.867 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836543 2024-09-13T15:59:28.867 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:28.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:59:28.869 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836543 2024-09-13T15:59:28.869 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:28.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836543 2024-09-13T15:59:28.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-09-13T15:59:28.870 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836543 2024-09-13T15:59:28.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:29.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836542 -lt 21474836543 2024-09-13T15:59:29.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:59:30.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:59:30.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:30.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836542 -lt 21474836543 2024-09-13T15:59:30.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:59:31.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T15:59:31.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:31.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836544 -lt 21474836543 2024-09-13T15:59:31.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:31.816 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:31.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673019 2024-09-13T15:59:31.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:59:31.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673019 2024-09-13T15:59:31.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:31.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673019 2024-09-13T15:59:31.819 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673019 2024-09-13T15:59:31.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673019' 2024-09-13T15:59:31.820 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:59:32.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673019 -lt 42949673019 2024-09-13T15:59:32.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:32.142 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:32.142 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509496 2024-09-13T15:59:32.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:59:32.145 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509496 2024-09-13T15:59:32.145 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:32.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509496 2024-09-13T15:59:32.146 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509496 2024-09-13T15:59:32.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2024-09-13T15:59:32.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:32.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509496 -lt 64424509496 2024-09-13T15:59:32.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T15:59:32.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T15:59:32.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T15:59:32.808 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:59:32.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T15:59:32.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T15:59:32.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T15:59:32.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:59:32.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:59:33.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:59:33.138 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:59:33.138 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:59:33.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:59:33.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:33.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:59:33.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836545 2024-09-13T15:59:33.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836545 2024-09-13T15:59:33.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545' 2024-09-13T15:59:33.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:33.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:59:33.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673021 2024-09-13T15:59:33.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673021 2024-09-13T15:59:33.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545 1-42949673021' 2024-09-13T15:59:33.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:33.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:59:33.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509498 2024-09-13T15:59:33.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509498 2024-09-13T15:59:33.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545 1-42949673021 2-64424509498' 2024-09-13T15:59:33.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:33.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836545 2024-09-13T15:59:33.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:33.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:59:33.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836545 2024-09-13T15:59:33.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:33.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836545 2024-09-13T15:59:33.597 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836545 2024-09-13T15:59:33.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836545' 2024-09-13T15:59:33.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:33.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836544 -lt 21474836545 2024-09-13T15:59:33.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:59:34.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:59:34.946 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:35.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836545 -lt 21474836545 2024-09-13T15:59:35.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:35.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673021 2024-09-13T15:59:35.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:35.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:59:35.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673021 2024-09-13T15:59:35.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:35.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673021 2024-09-13T15:59:35.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673021' 2024-09-13T15:59:35.275 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673021 2024-09-13T15:59:35.275 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:59:35.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673021 -lt 42949673021 2024-09-13T15:59:35.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:35.651 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509498 2024-09-13T15:59:35.651 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:35.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:59:35.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509498 2024-09-13T15:59:35.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:35.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509498 2024-09-13T15:59:35.656 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509498 2024-09-13T15:59:35.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509498' 2024-09-13T15:59:35.656 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:35.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509498 -lt 64424509498 2024-09-13T15:59:35.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T15:59:35.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T15:59:35.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T15:59:36.252 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:59:36.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T15:59:36.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T15:59:36.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T15:59:36.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:59:36.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:59:36.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:59:36.565 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:59:36.565 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:59:36.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:59:36.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:36.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:59:36.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836547 2024-09-13T15:59:36.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836547 2024-09-13T15:59:36.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547' 2024-09-13T15:59:36.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:36.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:59:36.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673023 2024-09-13T15:59:36.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673023 2024-09-13T15:59:36.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-42949673023' 2024-09-13T15:59:36.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:36.837 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:59:37.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509500 2024-09-13T15:59:37.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509500 2024-09-13T15:59:37.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-42949673023 2-64424509500' 2024-09-13T15:59:37.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:37.000 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836547 2024-09-13T15:59:37.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:37.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:59:37.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836547 2024-09-13T15:59:37.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:37.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836547 2024-09-13T15:59:37.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-09-13T15:59:37.004 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836547 2024-09-13T15:59:37.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:37.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836546 -lt 21474836547 2024-09-13T15:59:37.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:59:38.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:59:38.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:38.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836547 -lt 21474836547 2024-09-13T15:59:38.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:38.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673023 2024-09-13T15:59:38.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:38.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:59:38.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:38.620 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673023 2024-09-13T15:59:38.621 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673023 2024-09-13T15:59:38.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673023 2024-09-13T15:59:38.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2024-09-13T15:59:38.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:59:38.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673023 -lt 42949673023 2024-09-13T15:59:38.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:38.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509500 2024-09-13T15:59:38.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:38.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:59:38.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:38.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509500 2024-09-13T15:59:38.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509500 2024-09-13T15:59:38.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509500' 2024-09-13T15:59:38.985 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509500 2024-09-13T15:59:38.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:39.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509500 -lt 64424509500 2024-09-13T15:59:39.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T15:59:39.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T15:59:39.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T15:59:39.694 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:59:39.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T15:59:39.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T15:59:39.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T15:59:39.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:59:39.714 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:59:40.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:59:40.023 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:59:40.023 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:59:40.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:59:40.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:40.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:59:40.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836548 2024-09-13T15:59:40.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836548 2024-09-13T15:59:40.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548' 2024-09-13T15:59:40.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:40.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:59:40.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673024 2024-09-13T15:59:40.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673024 2024-09-13T15:59:40.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-42949673024' 2024-09-13T15:59:40.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:40.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:59:40.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509501 2024-09-13T15:59:40.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509501 2024-09-13T15:59:40.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-42949673024 2-64424509501' 2024-09-13T15:59:40.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:40.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836548 2024-09-13T15:59:40.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:40.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:59:40.432 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836548 2024-09-13T15:59:40.432 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:40.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836548 2024-09-13T15:59:40.433 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836548 2024-09-13T15:59:40.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-09-13T15:59:40.434 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:40.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836549 -lt 21474836548 2024-09-13T15:59:40.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:40.780 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673024 2024-09-13T15:59:40.780 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:40.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:59:40.782 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673024 2024-09-13T15:59:40.782 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:40.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673024 2024-09-13T15:59:40.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673024' 2024-09-13T15:59:40.783 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673024 2024-09-13T15:59:40.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:59:41.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673024 -lt 42949673024 2024-09-13T15:59:41.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:41.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509501 2024-09-13T15:59:41.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:41.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:59:41.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:41.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509501 2024-09-13T15:59:41.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509501 2024-09-13T15:59:41.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509501' 2024-09-13T15:59:41.094 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509501 2024-09-13T15:59:41.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:41.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509500 -lt 64424509501 2024-09-13T15:59:41.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:59:42.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:59:42.466 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:42.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509502 -lt 64424509501 2024-09-13T15:59:42.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T15:59:42.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T15:59:42.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T15:59:43.112 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:59:43.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T15:59:43.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T15:59:43.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T15:59:43.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:59:43.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:59:43.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:59:43.427 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:59:43.427 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:59:43.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:59:43.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:43.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:59:43.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836550 2024-09-13T15:59:43.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836550 2024-09-13T15:59:43.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550' 2024-09-13T15:59:43.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:43.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:59:43.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673026 2024-09-13T15:59:43.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673026 2024-09-13T15:59:43.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673026' 2024-09-13T15:59:43.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:43.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:59:43.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509503 2024-09-13T15:59:43.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509503 2024-09-13T15:59:43.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673026 2-64424509503' 2024-09-13T15:59:43.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:43.880 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836550 2024-09-13T15:59:43.880 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:43.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:59:43.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836550 2024-09-13T15:59:43.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:43.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836550 2024-09-13T15:59:43.882 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836550 2024-09-13T15:59:43.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-09-13T15:59:43.883 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:44.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836549 -lt 21474836550 2024-09-13T15:59:44.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:59:45.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:59:45.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:45.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836550 -lt 21474836550 2024-09-13T15:59:45.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:45.486 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673026 2024-09-13T15:59:45.487 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:45.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:59:45.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:45.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673026 2024-09-13T15:59:45.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673026 2024-09-13T15:59:45.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673026' 2024-09-13T15:59:45.491 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673026 2024-09-13T15:59:45.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:59:45.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673026 -lt 42949673026 2024-09-13T15:59:45.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:45.857 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509503 2024-09-13T15:59:45.857 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:45.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:59:45.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509503 2024-09-13T15:59:45.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:45.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509503 2024-09-13T15:59:45.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2024-09-13T15:59:45.861 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509503 2024-09-13T15:59:45.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:46.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509503 -lt 64424509503 2024-09-13T15:59:46.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T15:59:46.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T15:59:46.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T15:59:46.568 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:59:46.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T15:59:46.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T15:59:46.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T15:59:46.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:59:46.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:59:46.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:59:46.939 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:59:46.939 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:59:46.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:59:46.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:46.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:59:47.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836552 2024-09-13T15:59:47.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836552 2024-09-13T15:59:47.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552' 2024-09-13T15:59:47.082 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:47.082 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:59:47.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673028 2024-09-13T15:59:47.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673028 2024-09-13T15:59:47.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-42949673028' 2024-09-13T15:59:47.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:47.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:59:47.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509505 2024-09-13T15:59:47.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509505 2024-09-13T15:59:47.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-42949673028 2-64424509505' 2024-09-13T15:59:47.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:47.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836552 2024-09-13T15:59:47.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:47.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:59:47.349 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836552 2024-09-13T15:59:47.349 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:47.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836552 2024-09-13T15:59:47.350 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836552 2024-09-13T15:59:47.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836552' 2024-09-13T15:59:47.350 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:47.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836551 -lt 21474836552 2024-09-13T15:59:47.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:59:48.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:59:48.650 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:48.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836552 -lt 21474836552 2024-09-13T15:59:48.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:48.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673028 2024-09-13T15:59:48.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:48.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:59:48.969 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673028 2024-09-13T15:59:48.969 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:48.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673028 2024-09-13T15:59:48.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673028' 2024-09-13T15:59:48.971 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673028 2024-09-13T15:59:48.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:59:49.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673028 -lt 42949673028 2024-09-13T15:59:49.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:49.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509505 2024-09-13T15:59:49.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:49.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:59:49.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509505 2024-09-13T15:59:49.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:49.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509505 2024-09-13T15:59:49.270 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509505 2024-09-13T15:59:49.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2024-09-13T15:59:49.270 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:49.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509505 -lt 64424509505 2024-09-13T15:59:49.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T15:59:49.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T15:59:49.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T15:59:49.905 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:59:49.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T15:59:49.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T15:59:49.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T15:59:49.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:59:49.924 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:59:50.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:59:50.237 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:59:50.237 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:59:50.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:59:50.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:50.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:59:50.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836554 2024-09-13T15:59:50.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836554 2024-09-13T15:59:50.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554' 2024-09-13T15:59:50.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:50.420 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:59:50.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673029 2024-09-13T15:59:50.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673029 2024-09-13T15:59:50.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673029' 2024-09-13T15:59:50.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:50.557 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:59:50.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509506 2024-09-13T15:59:50.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509506 2024-09-13T15:59:50.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673029 2-64424509506' 2024-09-13T15:59:50.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:50.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836554 2024-09-13T15:59:50.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:50.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:59:50.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836554 2024-09-13T15:59:50.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:50.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836554 2024-09-13T15:59:50.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2024-09-13T15:59:50.687 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836554 2024-09-13T15:59:50.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:51.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836553 -lt 21474836554 2024-09-13T15:59:51.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:59:52.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:59:52.012 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:52.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836553 -lt 21474836554 2024-09-13T15:59:52.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:59:53.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T15:59:53.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:53.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836554 -lt 21474836554 2024-09-13T15:59:53.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:53.675 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673029 2024-09-13T15:59:53.675 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:53.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:59:53.677 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673029 2024-09-13T15:59:53.677 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:53.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673029 2024-09-13T15:59:53.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673029' 2024-09-13T15:59:53.679 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673029 2024-09-13T15:59:53.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:59:53.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673030 -lt 42949673029 2024-09-13T15:59:53.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:53.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509506 2024-09-13T15:59:53.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:53.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:59:53.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509506 2024-09-13T15:59:53.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:53.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509506 2024-09-13T15:59:53.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509506' 2024-09-13T15:59:53.989 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509506 2024-09-13T15:59:53.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:54.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509507 -lt 64424509506 2024-09-13T15:59:54.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T15:59:54.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T15:59:54.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T15:59:54.731 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:59:54.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T15:59:54.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T15:59:54.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T15:59:54.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:59:54.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:59:55.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:59:55.073 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:59:55.073 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:59:55.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:59:55.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:55.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:59:55.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836555 2024-09-13T15:59:55.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836555 2024-09-13T15:59:55.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555' 2024-09-13T15:59:55.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:55.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:59:55.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673031 2024-09-13T15:59:55.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673031 2024-09-13T15:59:55.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-42949673031' 2024-09-13T15:59:55.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:55.359 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:59:55.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509508 2024-09-13T15:59:55.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509508 2024-09-13T15:59:55.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509508' 2024-09-13T15:59:55.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:55.492 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836555 2024-09-13T15:59:55.492 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:55.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:59:55.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836555 2024-09-13T15:59:55.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:55.494 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836555 2024-09-13T15:59:55.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836555 2024-09-13T15:59:55.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836555' 2024-09-13T15:59:55.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:55.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836554 -lt 21474836555 2024-09-13T15:59:55.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T15:59:56.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T15:59:56.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:57.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836556 -lt 21474836555 2024-09-13T15:59:57.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:57.269 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673031 2024-09-13T15:59:57.269 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:57.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T15:59:57.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673031 2024-09-13T15:59:57.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:57.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673031 2024-09-13T15:59:57.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2024-09-13T15:59:57.272 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673031 2024-09-13T15:59:57.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T15:59:57.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673032 -lt 42949673031 2024-09-13T15:59:57.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:57.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509508 2024-09-13T15:59:57.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:57.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T15:59:57.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509508 2024-09-13T15:59:57.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:57.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509508 2024-09-13T15:59:57.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509508' 2024-09-13T15:59:57.637 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509508 2024-09-13T15:59:57.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T15:59:57.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509508 -lt 64424509508 2024-09-13T15:59:57.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T15:59:57.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T15:59:57.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T15:59:58.276 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T15:59:58.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T15:59:58.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T15:59:58.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T15:59:58.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T15:59:58.294 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T15:59:58.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T15:59:58.601 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T15:59:58.601 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T15:59:58.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T15:59:58.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:58.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T15:59:58.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836557 2024-09-13T15:59:58.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836557 2024-09-13T15:59:58.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836557' 2024-09-13T15:59:58.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:58.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T15:59:58.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673033 2024-09-13T15:59:58.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673033 2024-09-13T15:59:58.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836557 1-42949673033' 2024-09-13T15:59:58.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T15:59:58.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T15:59:59.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509510 2024-09-13T15:59:59.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509510 2024-09-13T15:59:59.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836557 1-42949673033 2-64424509510' 2024-09-13T15:59:59.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T15:59:59.037 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836557 2024-09-13T15:59:59.037 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T15:59:59.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T15:59:59.038 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836557 2024-09-13T15:59:59.038 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T15:59:59.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836557 2024-09-13T15:59:59.040 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836557 2024-09-13T15:59:59.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836557' 2024-09-13T15:59:59.040 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T15:59:59.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836556 -lt 21474836557 2024-09-13T15:59:59.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:00.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:00:00.339 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:00.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836558 -lt 21474836557 2024-09-13T16:00:00.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:00.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673033 2024-09-13T16:00:00.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:00.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:00.691 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673033 2024-09-13T16:00:00.691 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:00.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673033 2024-09-13T16:00:00.693 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673033 2024-09-13T16:00:00.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673033' 2024-09-13T16:00:00.693 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:01.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673033 -lt 42949673033 2024-09-13T16:00:01.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:01.028 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509510 2024-09-13T16:00:01.029 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:01.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:01.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509510 2024-09-13T16:00:01.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:01.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509510 2024-09-13T16:00:01.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-09-13T16:00:01.033 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509510 2024-09-13T16:00:01.033 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:01.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509510 -lt 64424509510 2024-09-13T16:00:01.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:01.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:01.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:01.612 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:01.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:01.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:01.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:01.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:01.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:01.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:01.938 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:01.938 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:01.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:01.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:01.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:02.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836559 2024-09-13T16:00:02.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836559 2024-09-13T16:00:02.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559' 2024-09-13T16:00:02.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:02.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:02.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673035 2024-09-13T16:00:02.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673035 2024-09-13T16:00:02.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673035' 2024-09-13T16:00:02.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:02.201 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:02.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509512 2024-09-13T16:00:02.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509512 2024-09-13T16:00:02.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673035 2-64424509512' 2024-09-13T16:00:02.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:02.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836559 2024-09-13T16:00:02.326 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:02.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:02.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836559 2024-09-13T16:00:02.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:02.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836559 2024-09-13T16:00:02.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836559' 2024-09-13T16:00:02.330 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836559 2024-09-13T16:00:02.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:02.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836559 -lt 21474836559 2024-09-13T16:00:02.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:02.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673035 2024-09-13T16:00:02.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:02.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:02.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673035 2024-09-13T16:00:02.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:02.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673035 2024-09-13T16:00:02.627 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673035 2024-09-13T16:00:02.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-09-13T16:00:02.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:02.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673035 -lt 42949673035 2024-09-13T16:00:02.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:02.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509512 2024-09-13T16:00:02.978 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:02.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:02.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509512 2024-09-13T16:00:02.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:02.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509512 2024-09-13T16:00:02.983 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509512 2024-09-13T16:00:02.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2024-09-13T16:00:02.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:03.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509512 -lt 64424509512 2024-09-13T16:00:03.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:03.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:03.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:03.655 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:03.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:03.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:03.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:03.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:03.671 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:03.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:03.990 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:03.990 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:03.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:03.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:03.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:04.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836560 2024-09-13T16:00:04.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836560 2024-09-13T16:00:04.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560' 2024-09-13T16:00:04.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:04.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:04.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673036 2024-09-13T16:00:04.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673036 2024-09-13T16:00:04.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673036' 2024-09-13T16:00:04.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:04.251 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:04.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509513 2024-09-13T16:00:04.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509513 2024-09-13T16:00:04.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673036 2-64424509513' 2024-09-13T16:00:04.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:04.381 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836560 2024-09-13T16:00:04.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:04.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:04.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836560 2024-09-13T16:00:04.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:04.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836560 2024-09-13T16:00:04.386 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836560 2024-09-13T16:00:04.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836560' 2024-09-13T16:00:04.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:04.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836560 -lt 21474836560 2024-09-13T16:00:04.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:04.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:04.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673036 2024-09-13T16:00:04.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:04.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:04.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673036 2024-09-13T16:00:04.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673036 2024-09-13T16:00:04.712 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673036 2024-09-13T16:00:04.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673036' 2024-09-13T16:00:04.712 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:05.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673036 -lt 42949673036 2024-09-13T16:00:05.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:05.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509513 2024-09-13T16:00:05.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:05.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:05.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509513 2024-09-13T16:00:05.020 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:05.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509513 2024-09-13T16:00:05.021 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509513 2024-09-13T16:00:05.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509513' 2024-09-13T16:00:05.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:05.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509513 -lt 64424509513 2024-09-13T16:00:05.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:05.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:05.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:05.665 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:05.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:05.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:05.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:05.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:05.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:06.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:06.001 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:06.002 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:06.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:06.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:06.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:06.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836562 2024-09-13T16:00:06.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836562 2024-09-13T16:00:06.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562' 2024-09-13T16:00:06.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:06.190 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:06.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673038 2024-09-13T16:00:06.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673038 2024-09-13T16:00:06.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673038' 2024-09-13T16:00:06.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:06.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:06.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509514 2024-09-13T16:00:06.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509514 2024-09-13T16:00:06.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673038 2-64424509514' 2024-09-13T16:00:06.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:06.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836562 2024-09-13T16:00:06.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:06.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:06.470 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836562 2024-09-13T16:00:06.471 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:06.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836562 2024-09-13T16:00:06.472 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836562 2024-09-13T16:00:06.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836562' 2024-09-13T16:00:06.472 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:06.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836562 -lt 21474836562 2024-09-13T16:00:06.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:06.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673038 2024-09-13T16:00:06.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:06.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:06.779 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673038 2024-09-13T16:00:06.779 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:06.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673038 2024-09-13T16:00:06.780 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673038 2024-09-13T16:00:06.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-09-13T16:00:06.781 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:07.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673038 -lt 42949673038 2024-09-13T16:00:07.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:07.130 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509514 2024-09-13T16:00:07.130 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:07.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:07.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509514 2024-09-13T16:00:07.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:07.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509514 2024-09-13T16:00:07.134 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509514 2024-09-13T16:00:07.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-09-13T16:00:07.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:07.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509513 -lt 64424509514 2024-09-13T16:00:07.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:08.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:00:08.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:08.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509515 -lt 64424509514 2024-09-13T16:00:08.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:08.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:08.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:09.130 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:09.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:09.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:09.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:09.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:09.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:09.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:09.453 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:09.453 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:09.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:09.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:09.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:09.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836563 2024-09-13T16:00:09.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836563 2024-09-13T16:00:09.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563' 2024-09-13T16:00:09.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:09.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:09.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673039 2024-09-13T16:00:09.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673039 2024-09-13T16:00:09.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673039' 2024-09-13T16:00:09.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:09.828 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:09.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509516 2024-09-13T16:00:09.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509516 2024-09-13T16:00:09.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673039 2-64424509516' 2024-09-13T16:00:09.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:09.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836563 2024-09-13T16:00:09.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:09.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:09.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836563 2024-09-13T16:00:09.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:09.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836563 2024-09-13T16:00:09.978 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836563 2024-09-13T16:00:09.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836563' 2024-09-13T16:00:09.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:10.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836562 -lt 21474836563 2024-09-13T16:00:10.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:11.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:00:11.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:11.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836564 -lt 21474836563 2024-09-13T16:00:11.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:11.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673039 2024-09-13T16:00:11.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:11.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:11.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673039 2024-09-13T16:00:11.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:11.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673039 2024-09-13T16:00:11.618 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673039 2024-09-13T16:00:11.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673039' 2024-09-13T16:00:11.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:11.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673039 -lt 42949673039 2024-09-13T16:00:11.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:11.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509516 2024-09-13T16:00:11.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:11.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:11.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509516 2024-09-13T16:00:11.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:11.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509516 2024-09-13T16:00:11.989 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509516 2024-09-13T16:00:11.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509516' 2024-09-13T16:00:11.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:12.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509516 -lt 64424509516 2024-09-13T16:00:12.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:12.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:12.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:12.772 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:12.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:12.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:12.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:12.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:12.791 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:13.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:13.117 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:13.117 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:13.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:13.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:13.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:13.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836565 2024-09-13T16:00:13.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836565 2024-09-13T16:00:13.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565' 2024-09-13T16:00:13.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:13.263 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:13.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673041 2024-09-13T16:00:13.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673041 2024-09-13T16:00:13.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673041' 2024-09-13T16:00:13.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:13.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:13.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509518 2024-09-13T16:00:13.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509518 2024-09-13T16:00:13.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509518' 2024-09-13T16:00:13.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:13.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836565 2024-09-13T16:00:13.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:13.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:13.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836565 2024-09-13T16:00:13.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:13.532 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836565 2024-09-13T16:00:13.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836565 2024-09-13T16:00:13.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-09-13T16:00:13.533 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:13.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836564 -lt 21474836565 2024-09-13T16:00:13.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:14.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:00:14.857 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:15.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836565 -lt 21474836565 2024-09-13T16:00:15.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:15.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673041 2024-09-13T16:00:15.184 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:15.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:15.186 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673041 2024-09-13T16:00:15.186 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:15.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673041 2024-09-13T16:00:15.187 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673041 2024-09-13T16:00:15.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673041' 2024-09-13T16:00:15.188 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:15.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673041 -lt 42949673041 2024-09-13T16:00:15.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:15.540 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509518 2024-09-13T16:00:15.540 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:15.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:15.542 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509518 2024-09-13T16:00:15.543 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:15.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509518 2024-09-13T16:00:15.544 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509518 2024-09-13T16:00:15.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2024-09-13T16:00:15.544 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:15.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509518 -lt 64424509518 2024-09-13T16:00:15.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:15.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:15.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:16.171 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:16.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:16.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:16.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:16.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:16.188 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:16.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:16.506 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:16.507 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:16.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:16.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:16.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:16.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836567 2024-09-13T16:00:16.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836567 2024-09-13T16:00:16.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567' 2024-09-13T16:00:16.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:16.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:16.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673043 2024-09-13T16:00:16.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673043 2024-09-13T16:00:16.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673043' 2024-09-13T16:00:16.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:16.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:16.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509520 2024-09-13T16:00:16.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509520 2024-09-13T16:00:16.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509520' 2024-09-13T16:00:16.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:16.964 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836567 2024-09-13T16:00:16.964 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:16.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:16.965 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836567 2024-09-13T16:00:16.965 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:16.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836567 2024-09-13T16:00:16.967 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836567 2024-09-13T16:00:16.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2024-09-13T16:00:16.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:17.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836566 -lt 21474836567 2024-09-13T16:00:17.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:18.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:00:18.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:18.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836567 -lt 21474836567 2024-09-13T16:00:18.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:18.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673043 2024-09-13T16:00:18.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:18.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:18.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673043 2024-09-13T16:00:18.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673043 2024-09-13T16:00:18.609 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673043 2024-09-13T16:00:18.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-09-13T16:00:18.609 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:18.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673043 -lt 42949673043 2024-09-13T16:00:18.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:18.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509520 2024-09-13T16:00:18.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:18.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:18.974 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509520 2024-09-13T16:00:18.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:18.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509520 2024-09-13T16:00:18.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-09-13T16:00:18.976 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509520 2024-09-13T16:00:18.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:19.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509520 -lt 64424509520 2024-09-13T16:00:19.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:19.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:19.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:19.637 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:19.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:19.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:19.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:19.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:19.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:19.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:19.980 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:19.980 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:19.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:19.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:19.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:20.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836568 2024-09-13T16:00:20.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836568 2024-09-13T16:00:20.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568' 2024-09-13T16:00:20.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:20.127 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:20.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673044 2024-09-13T16:00:20.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673044 2024-09-13T16:00:20.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568 1-42949673044' 2024-09-13T16:00:20.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:20.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:20.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509521 2024-09-13T16:00:20.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509521 2024-09-13T16:00:20.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568 1-42949673044 2-64424509521' 2024-09-13T16:00:20.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:20.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836568 2024-09-13T16:00:20.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:20.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:20.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836568 2024-09-13T16:00:20.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:20.403 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836568 2024-09-13T16:00:20.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836568 2024-09-13T16:00:20.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2024-09-13T16:00:20.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:20.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836569 -lt 21474836568 2024-09-13T16:00:20.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:20.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673044 2024-09-13T16:00:20.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:20.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:20.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673044 2024-09-13T16:00:20.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:20.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673044 2024-09-13T16:00:20.743 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673044 2024-09-13T16:00:20.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673044' 2024-09-13T16:00:20.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:21.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673044 -lt 42949673044 2024-09-13T16:00:21.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:21.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509521 2024-09-13T16:00:21.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:21.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:21.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509521 2024-09-13T16:00:21.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:21.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509521 2024-09-13T16:00:21.070 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509521 2024-09-13T16:00:21.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-09-13T16:00:21.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:21.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509521 -lt 64424509521 2024-09-13T16:00:21.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:21.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:21.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:21.726 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:21.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:21.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:21.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:21.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:21.745 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:22.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:22.045 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:22.045 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:22.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:22.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:22.046 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:22.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836570 2024-09-13T16:00:22.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836570 2024-09-13T16:00:22.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836570' 2024-09-13T16:00:22.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:22.240 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:22.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673046 2024-09-13T16:00:22.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673046 2024-09-13T16:00:22.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836570 1-42949673046' 2024-09-13T16:00:22.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:22.383 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:22.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509523 2024-09-13T16:00:22.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509523 2024-09-13T16:00:22.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836570 1-42949673046 2-64424509523' 2024-09-13T16:00:22.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:22.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836570 2024-09-13T16:00:22.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:22.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:22.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836570 2024-09-13T16:00:22.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:22.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836570 2024-09-13T16:00:22.527 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836570 2024-09-13T16:00:22.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2024-09-13T16:00:22.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:22.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836570 -lt 21474836570 2024-09-13T16:00:22.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:22.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673046 2024-09-13T16:00:22.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:22.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:22.836 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673046 2024-09-13T16:00:22.836 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:22.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673046 2024-09-13T16:00:22.837 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673046 2024-09-13T16:00:22.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-09-13T16:00:22.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:23.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673046 -lt 42949673046 2024-09-13T16:00:23.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:23.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509523 2024-09-13T16:00:23.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:23.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:23.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509523 2024-09-13T16:00:23.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:23.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509523 2024-09-13T16:00:23.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509523' 2024-09-13T16:00:23.220 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509523 2024-09-13T16:00:23.220 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:23.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509522 -lt 64424509523 2024-09-13T16:00:23.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:24.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:00:24.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:24.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509523 -lt 64424509523 2024-09-13T16:00:24.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:24.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:24.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:25.127 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:25.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:25.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:25.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:25.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:25.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:25.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:25.451 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:25.451 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:25.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:25.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:25.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:25.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836572 2024-09-13T16:00:25.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836572 2024-09-13T16:00:25.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572' 2024-09-13T16:00:25.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:25.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:25.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673047 2024-09-13T16:00:25.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673047 2024-09-13T16:00:25.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673047' 2024-09-13T16:00:25.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:25.753 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:25.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509524 2024-09-13T16:00:25.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509524 2024-09-13T16:00:25.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673047 2-64424509524' 2024-09-13T16:00:25.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:25.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836572 2024-09-13T16:00:25.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:25.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:25.883 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836572 2024-09-13T16:00:25.883 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:25.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836572 2024-09-13T16:00:25.884 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836572 2024-09-13T16:00:25.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-09-13T16:00:25.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:26.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836570 -lt 21474836572 2024-09-13T16:00:26.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:27.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:00:27.189 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:27.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836572 -lt 21474836572 2024-09-13T16:00:27.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:27.504 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673047 2024-09-13T16:00:27.504 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:27.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:27.506 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673047 2024-09-13T16:00:27.506 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:27.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673047 2024-09-13T16:00:27.508 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673047 2024-09-13T16:00:27.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673047' 2024-09-13T16:00:27.508 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:27.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673048 -lt 42949673047 2024-09-13T16:00:27.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:27.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:27.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509524 2024-09-13T16:00:27.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:27.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509524 2024-09-13T16:00:27.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:27.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509524 2024-09-13T16:00:27.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-09-13T16:00:27.819 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509524 2024-09-13T16:00:27.819 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:28.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509524 -lt 64424509524 2024-09-13T16:00:28.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:28.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:28.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:28.425 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:28.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:28.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:28.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:28.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:28.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:28.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:28.756 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:28.757 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:28.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:28.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:28.757 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:28.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836573 2024-09-13T16:00:28.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836573 2024-09-13T16:00:28.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573' 2024-09-13T16:00:28.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:28.893 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:29.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673049 2024-09-13T16:00:29.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673049 2024-09-13T16:00:29.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673049' 2024-09-13T16:00:29.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:29.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:29.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509526 2024-09-13T16:00:29.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509526 2024-09-13T16:00:29.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673049 2-64424509526' 2024-09-13T16:00:29.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:29.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836573 2024-09-13T16:00:29.169 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:29.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:29.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836573 2024-09-13T16:00:29.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:29.171 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836573 2024-09-13T16:00:29.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836573 2024-09-13T16:00:29.172 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-09-13T16:00:29.172 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:29.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836572 -lt 21474836573 2024-09-13T16:00:29.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:30.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:00:30.474 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:30.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836574 -lt 21474836573 2024-09-13T16:00:30.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:30.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673049 2024-09-13T16:00:30.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:30.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:30.792 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673049 2024-09-13T16:00:30.792 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:30.794 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673049 2024-09-13T16:00:30.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673049 2024-09-13T16:00:30.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-09-13T16:00:30.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:31.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673049 -lt 42949673049 2024-09-13T16:00:31.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:31.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509526 2024-09-13T16:00:31.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:31.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:31.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509526 2024-09-13T16:00:31.168 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:31.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509526 2024-09-13T16:00:31.169 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509526 2024-09-13T16:00:31.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-09-13T16:00:31.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:31.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509526 -lt 64424509526 2024-09-13T16:00:31.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:31.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:31.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:31.791 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:31.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:31.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:31.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:31.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:31.809 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:32.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:32.117 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:32.117 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:32.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:32.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:32.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:32.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836575 2024-09-13T16:00:32.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836575 2024-09-13T16:00:32.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836575' 2024-09-13T16:00:32.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:32.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:32.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673051 2024-09-13T16:00:32.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673051 2024-09-13T16:00:32.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836575 1-42949673051' 2024-09-13T16:00:32.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:32.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:32.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509528 2024-09-13T16:00:32.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509528 2024-09-13T16:00:32.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836575 1-42949673051 2-64424509528' 2024-09-13T16:00:32.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:32.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836575 2024-09-13T16:00:32.529 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:32.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:32.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836575 2024-09-13T16:00:32.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:32.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836575 2024-09-13T16:00:32.533 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836575 2024-09-13T16:00:32.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2024-09-13T16:00:32.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:32.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836575 -lt 21474836575 2024-09-13T16:00:32.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:32.858 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673051 2024-09-13T16:00:32.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:32.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:32.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673051 2024-09-13T16:00:32.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:32.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673051 2024-09-13T16:00:32.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-09-13T16:00:32.862 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673051 2024-09-13T16:00:32.862 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:33.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673051 -lt 42949673051 2024-09-13T16:00:33.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:33.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509528 2024-09-13T16:00:33.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:33.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:33.230 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:33.230 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509528 2024-09-13T16:00:33.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509528 2024-09-13T16:00:33.231 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509528 2024-09-13T16:00:33.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509528' 2024-09-13T16:00:33.232 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:33.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509527 -lt 64424509528 2024-09-13T16:00:33.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:34.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:00:34.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:34.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509528 -lt 64424509528 2024-09-13T16:00:34.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:34.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:34.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:35.176 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:35.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:35.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:35.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:35.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:35.195 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:35.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:35.505 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:35.505 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:35.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:35.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:35.506 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:35.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836577 2024-09-13T16:00:35.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836577 2024-09-13T16:00:35.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577' 2024-09-13T16:00:35.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:35.646 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:35.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673052 2024-09-13T16:00:35.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673052 2024-09-13T16:00:35.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673052' 2024-09-13T16:00:35.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:35.809 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:35.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509529 2024-09-13T16:00:35.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509529 2024-09-13T16:00:35.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673052 2-64424509529' 2024-09-13T16:00:35.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:35.954 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836577 2024-09-13T16:00:35.954 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:35.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:35.955 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836577 2024-09-13T16:00:35.955 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:35.956 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836577 2024-09-13T16:00:35.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836577 2024-09-13T16:00:35.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-09-13T16:00:35.956 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:36.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836575 -lt 21474836577 2024-09-13T16:00:36.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:37.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:00:37.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:37.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836577 -lt 21474836577 2024-09-13T16:00:37.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:37.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673052 2024-09-13T16:00:37.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:37.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:37.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673052 2024-09-13T16:00:37.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:37.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673052 2024-09-13T16:00:37.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-09-13T16:00:37.576 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673052 2024-09-13T16:00:37.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:37.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673053 -lt 42949673052 2024-09-13T16:00:37.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:37.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:37.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509529 2024-09-13T16:00:37.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:37.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509529 2024-09-13T16:00:37.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:37.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509529 2024-09-13T16:00:37.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-09-13T16:00:37.876 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509529 2024-09-13T16:00:37.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:38.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509529 -lt 64424509529 2024-09-13T16:00:38.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:38.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:38.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:38.488 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:38.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:38.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:38.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:38.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:38.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:38.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:38.858 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:38.858 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:38.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:38.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:38.858 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:39.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836578 2024-09-13T16:00:39.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836578 2024-09-13T16:00:39.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578' 2024-09-13T16:00:39.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:39.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:39.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673054 2024-09-13T16:00:39.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673054 2024-09-13T16:00:39.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673054' 2024-09-13T16:00:39.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:39.145 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:39.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509531 2024-09-13T16:00:39.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509531 2024-09-13T16:00:39.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836578 1-42949673054 2-64424509531' 2024-09-13T16:00:39.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:39.283 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836578 2024-09-13T16:00:39.283 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:39.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:39.285 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836578 2024-09-13T16:00:39.285 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:39.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836578 2024-09-13T16:00:39.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2024-09-13T16:00:39.286 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836578 2024-09-13T16:00:39.287 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:39.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836577 -lt 21474836578 2024-09-13T16:00:39.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:40.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:00:40.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:40.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836579 -lt 21474836578 2024-09-13T16:00:40.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:40.947 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673054 2024-09-13T16:00:40.947 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:40.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:40.950 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673054 2024-09-13T16:00:40.950 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:40.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673054 2024-09-13T16:00:40.952 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673054 2024-09-13T16:00:40.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-09-13T16:00:40.952 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:41.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673054 -lt 42949673054 2024-09-13T16:00:41.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:41.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509531 2024-09-13T16:00:41.337 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:41.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:41.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509531 2024-09-13T16:00:41.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:41.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509531 2024-09-13T16:00:41.342 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509531 2024-09-13T16:00:41.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2024-09-13T16:00:41.342 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:41.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509531 -lt 64424509531 2024-09-13T16:00:41.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:41.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:41.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:42.016 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:42.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:42.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:42.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:42.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:42.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:42.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:42.359 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:42.359 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:42.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:42.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:42.359 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:42.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836580 2024-09-13T16:00:42.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836580 2024-09-13T16:00:42.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580' 2024-09-13T16:00:42.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:42.503 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:42.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673056 2024-09-13T16:00:42.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673056 2024-09-13T16:00:42.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673056' 2024-09-13T16:00:42.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:42.639 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:42.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509533 2024-09-13T16:00:42.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509533 2024-09-13T16:00:42.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673056 2-64424509533' 2024-09-13T16:00:42.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:42.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836580 2024-09-13T16:00:42.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:42.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:42.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836580 2024-09-13T16:00:42.776 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:42.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836580 2024-09-13T16:00:42.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-09-13T16:00:42.777 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836580 2024-09-13T16:00:42.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:43.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836579 -lt 21474836580 2024-09-13T16:00:43.084 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:44.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:00:44.086 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:44.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836579 -lt 21474836580 2024-09-13T16:00:44.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:45.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T16:00:45.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:45.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836580 -lt 21474836580 2024-09-13T16:00:45.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:45.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673056 2024-09-13T16:00:45.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:45.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:45.747 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673056 2024-09-13T16:00:45.747 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:45.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673056 2024-09-13T16:00:45.748 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673056 2024-09-13T16:00:45.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-09-13T16:00:45.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:46.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673056 -lt 42949673056 2024-09-13T16:00:46.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:46.060 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:46.061 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509533 2024-09-13T16:00:46.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:46.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509533 2024-09-13T16:00:46.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:46.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509533 2024-09-13T16:00:46.064 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509533 2024-09-13T16:00:46.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2024-09-13T16:00:46.065 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:46.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509533 -lt 64424509533 2024-09-13T16:00:46.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:46.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:46.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:46.661 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:46.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:46.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:46.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:46.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:46.679 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:47.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:47.043 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:47.043 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:47.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:47.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:47.044 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:47.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836582 2024-09-13T16:00:47.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836582 2024-09-13T16:00:47.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582' 2024-09-13T16:00:47.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:47.187 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:47.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673058 2024-09-13T16:00:47.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673058 2024-09-13T16:00:47.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582 1-42949673058' 2024-09-13T16:00:47.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:47.325 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:47.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509535 2024-09-13T16:00:47.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509535 2024-09-13T16:00:47.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582 1-42949673058 2-64424509535' 2024-09-13T16:00:47.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:47.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836582 2024-09-13T16:00:47.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:47.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:47.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836582 2024-09-13T16:00:47.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:47.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836582 2024-09-13T16:00:47.457 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836582 2024-09-13T16:00:47.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2024-09-13T16:00:47.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:47.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836581 -lt 21474836582 2024-09-13T16:00:47.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:48.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:00:48.757 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:49.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836582 -lt 21474836582 2024-09-13T16:00:49.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:49.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673058 2024-09-13T16:00:49.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:49.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:49.078 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673058 2024-09-13T16:00:49.078 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673058 2024-09-13T16:00:49.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673058' 2024-09-13T16:00:49.079 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673058 2024-09-13T16:00:49.080 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:49.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673058 -lt 42949673058 2024-09-13T16:00:49.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:49.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509535 2024-09-13T16:00:49.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:49.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:49.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509535 2024-09-13T16:00:49.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:49.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509535 2024-09-13T16:00:49.404 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509535 2024-09-13T16:00:49.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509535' 2024-09-13T16:00:49.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:49.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509535 -lt 64424509535 2024-09-13T16:00:49.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:49.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:49.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:49.989 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:50.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:50.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:50.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:50.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:50.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:50.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:50.306 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:50.306 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:50.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:50.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:50.307 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:50.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836584 2024-09-13T16:00:50.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836584 2024-09-13T16:00:50.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584' 2024-09-13T16:00:50.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:50.441 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:50.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673059 2024-09-13T16:00:50.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673059 2024-09-13T16:00:50.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673059' 2024-09-13T16:00:50.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:50.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:50.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509536 2024-09-13T16:00:50.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509536 2024-09-13T16:00:50.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673059 2-64424509536' 2024-09-13T16:00:50.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:50.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836584 2024-09-13T16:00:50.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:50.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:50.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836584 2024-09-13T16:00:50.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:50.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836584 2024-09-13T16:00:50.763 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836584 2024-09-13T16:00:50.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-09-13T16:00:50.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:51.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836583 -lt 21474836584 2024-09-13T16:00:51.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:52.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:00:52.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:52.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836583 -lt 21474836584 2024-09-13T16:00:52.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:53.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T16:00:53.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:53.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836584 -lt 21474836584 2024-09-13T16:00:53.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:53.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673059 2024-09-13T16:00:53.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:53.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:53.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673059 2024-09-13T16:00:53.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:53.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673059 2024-09-13T16:00:53.710 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673059 2024-09-13T16:00:53.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-09-13T16:00:53.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:54.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673060 -lt 42949673059 2024-09-13T16:00:54.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:54.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509536 2024-09-13T16:00:54.086 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:54.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:54.088 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509536 2024-09-13T16:00:54.088 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:54.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509536 2024-09-13T16:00:54.089 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509536 2024-09-13T16:00:54.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-09-13T16:00:54.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:54.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509537 -lt 64424509536 2024-09-13T16:00:54.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:54.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:54.473 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:54.779 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:54.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:54.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:54.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:54.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:54.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:55.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:55.116 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:55.116 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:55.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:55.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:55.116 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:55.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836585 2024-09-13T16:00:55.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836585 2024-09-13T16:00:55.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585' 2024-09-13T16:00:55.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:55.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:55.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673061 2024-09-13T16:00:55.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673061 2024-09-13T16:00:55.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061' 2024-09-13T16:00:55.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:55.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:55.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509538 2024-09-13T16:00:55.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509538 2024-09-13T16:00:55.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509538' 2024-09-13T16:00:55.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:55.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836585 2024-09-13T16:00:55.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:55.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:55.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836585 2024-09-13T16:00:55.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:55.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836585 2024-09-13T16:00:55.527 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836585 2024-09-13T16:00:55.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-09-13T16:00:55.528 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:55.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836584 -lt 21474836585 2024-09-13T16:00:55.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:00:56.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:00:56.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:57.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836586 -lt 21474836585 2024-09-13T16:00:57.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:57.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673061 2024-09-13T16:00:57.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:57.176 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:00:57.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673061 2024-09-13T16:00:57.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:57.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673061 2024-09-13T16:00:57.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-09-13T16:00:57.179 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673061 2024-09-13T16:00:57.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:00:57.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673062 -lt 42949673061 2024-09-13T16:00:57.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:57.477 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509538 2024-09-13T16:00:57.477 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:57.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:00:57.480 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509538 2024-09-13T16:00:57.480 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:57.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509538 2024-09-13T16:00:57.485 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509538 2024-09-13T16:00:57.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509538' 2024-09-13T16:00:57.485 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:00:57.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509538 -lt 64424509538 2024-09-13T16:00:57.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:00:57.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:00:57.784 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:00:58.073 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:00:58.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:00:58.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:00:58.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:00:58.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:00:58.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:00:58.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:00:58.397 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:00:58.397 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:00:58.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:00:58.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:58.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:00:58.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836587 2024-09-13T16:00:58.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836587 2024-09-13T16:00:58.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587' 2024-09-13T16:00:58.539 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:58.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:00:58.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673063 2024-09-13T16:00:58.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673063 2024-09-13T16:00:58.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673063' 2024-09-13T16:00:58.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:00:58.666 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:00:58.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509540 2024-09-13T16:00:58.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509540 2024-09-13T16:00:58.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673063 2-64424509540' 2024-09-13T16:00:58.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:00:58.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836587 2024-09-13T16:00:58.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:00:58.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:00:58.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836587 2024-09-13T16:00:58.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:00:58.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836587 2024-09-13T16:00:58.803 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836587 2024-09-13T16:00:58.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-09-13T16:00:58.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:00:59.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836586 -lt 21474836587 2024-09-13T16:00:59.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:01:00.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:01:00.115 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:01:00.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836586 -lt 21474836587 2024-09-13T16:01:00.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:01:01.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T16:01:01.445 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:01:01.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836588 -lt 21474836587 2024-09-13T16:01:01.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:01:01.773 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673063 2024-09-13T16:01:01.773 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:01:01.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:01:01.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:01:01.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673063 2024-09-13T16:01:01.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673063 2024-09-13T16:01:01.774 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673063 2024-09-13T16:01:01.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2024-09-13T16:01:01.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:01:02.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673063 -lt 42949673063 2024-09-13T16:01:02.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:01:02.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509540 2024-09-13T16:01:02.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:01:02.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:01:02.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509540 2024-09-13T16:01:02.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:01:02.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509540 2024-09-13T16:01:02.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509540' 2024-09-13T16:01:02.095 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509540 2024-09-13T16:01:02.096 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:01:02.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509540 -lt 64424509540 2024-09-13T16:01:02.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:01:02.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:01:02.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:01:02.774 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:01:02.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:01:02.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:01:02.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:01:02.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:01:02.790 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:01:03.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:01:03.112 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:01:03.112 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:01:03.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:01:03.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:01:03.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:01:03.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836589 2024-09-13T16:01:03.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836589 2024-09-13T16:01:03.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589' 2024-09-13T16:01:03.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:01:03.253 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:01:03.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673065 2024-09-13T16:01:03.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673065 2024-09-13T16:01:03.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673065' 2024-09-13T16:01:03.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:01:03.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:01:03.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509542 2024-09-13T16:01:03.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509542 2024-09-13T16:01:03.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673065 2-64424509542' 2024-09-13T16:01:03.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:01:03.518 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836589 2024-09-13T16:01:03.518 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:01:03.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:01:03.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836589 2024-09-13T16:01:03.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:01:03.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836589 2024-09-13T16:01:03.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-09-13T16:01:03.521 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836589 2024-09-13T16:01:03.521 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:01:03.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836588 -lt 21474836589 2024-09-13T16:01:03.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:01:04.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:01:04.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:01:05.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836589 -lt 21474836589 2024-09-13T16:01:05.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:01:05.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673065 2024-09-13T16:01:05.145 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:01:05.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:01:05.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673065 2024-09-13T16:01:05.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:01:05.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673065 2024-09-13T16:01:05.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673065' 2024-09-13T16:01:05.149 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673065 2024-09-13T16:01:05.149 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:01:05.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673065 -lt 42949673065 2024-09-13T16:01:05.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:01:05.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:01:05.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509542 2024-09-13T16:01:05.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:01:05.521 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509542 2024-09-13T16:01:05.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:01:05.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509542 2024-09-13T16:01:05.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509542' 2024-09-13T16:01:05.523 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509542 2024-09-13T16:01:05.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:01:05.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509542 -lt 64424509542 2024-09-13T16:01:05.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:01:05.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:01:05.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:01:06.114 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:01:06.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:01:06.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:01:06.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:01:06.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:01:06.131 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:01:06.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:01:06.439 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:01:06.439 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:01:06.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:01:06.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:01:06.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:01:06.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836591 2024-09-13T16:01:06.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836591 2024-09-13T16:01:06.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591' 2024-09-13T16:01:06.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:01:06.570 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:01:06.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673067 2024-09-13T16:01:06.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673067 2024-09-13T16:01:06.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067' 2024-09-13T16:01:06.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:01:06.712 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:01:06.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509544 2024-09-13T16:01:06.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509544 2024-09-13T16:01:06.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509544' 2024-09-13T16:01:06.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:01:06.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836591 2024-09-13T16:01:06.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:01:06.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:01:06.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836591 2024-09-13T16:01:06.842 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:01:06.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836591 2024-09-13T16:01:06.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2024-09-13T16:01:06.843 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836591 2024-09-13T16:01:06.844 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:01:07.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836590 -lt 21474836591 2024-09-13T16:01:07.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:01:08.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:01:08.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:01:08.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836590 -lt 21474836591 2024-09-13T16:01:08.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:01:09.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T16:01:09.470 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:01:09.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836591 -lt 21474836591 2024-09-13T16:01:09.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:01:09.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673067 2024-09-13T16:01:09.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:01:09.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:01:09.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673067 2024-09-13T16:01:09.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:01:09.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673067 2024-09-13T16:01:09.805 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673067 2024-09-13T16:01:09.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673067' 2024-09-13T16:01:09.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:01:10.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673067 -lt 42949673067 2024-09-13T16:01:10.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:01:10.196 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509544 2024-09-13T16:01:10.197 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:01:10.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:01:10.199 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509544 2024-09-13T16:01:10.199 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:01:10.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509544 2024-09-13T16:01:10.201 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509544 2024-09-13T16:01:10.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509544' 2024-09-13T16:01:10.202 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:01:10.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509544 -lt 64424509544 2024-09-13T16:01:10.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:01:10.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:01:10.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:01:10.820 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:01:10.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:01:10.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:01:10.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:01:10.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:01:10.838 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:01:11.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:01:11.164 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:01:11.165 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:01:11.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:01:11.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:01:11.165 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:01:11.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836593 2024-09-13T16:01:11.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836593 2024-09-13T16:01:11.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593' 2024-09-13T16:01:11.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:01:11.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:01:11.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673069 2024-09-13T16:01:11.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673069 2024-09-13T16:01:11.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673069' 2024-09-13T16:01:11.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:01:11.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:01:11.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509545 2024-09-13T16:01:11.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509545 2024-09-13T16:01:11.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673069 2-64424509545' 2024-09-13T16:01:11.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:01:11.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836593 2024-09-13T16:01:11.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:01:11.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:01:11.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836593 2024-09-13T16:01:11.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:01:11.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836593 2024-09-13T16:01:11.591 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836593 2024-09-13T16:01:11.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2024-09-13T16:01:11.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:01:11.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836592 -lt 21474836593 2024-09-13T16:01:11.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:01:12.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:01:12.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:01:13.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836593 -lt 21474836593 2024-09-13T16:01:13.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:01:13.210 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:01:13.210 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673069 2024-09-13T16:01:13.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:01:13.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673069 2024-09-13T16:01:13.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:01:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673069 2024-09-13T16:01:13.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673069' 2024-09-13T16:01:13.214 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673069 2024-09-13T16:01:13.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:01:13.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673069 -lt 42949673069 2024-09-13T16:01:13.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:01:13.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509545 2024-09-13T16:01:13.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:01:13.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:01:13.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509545 2024-09-13T16:01:13.522 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:01:13.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509545 2024-09-13T16:01:13.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509545' 2024-09-13T16:01:13.524 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509545 2024-09-13T16:01:13.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:01:13.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509546 -lt 64424509545 2024-09-13T16:01:13.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:01:13.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:01:13.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:01:14.145 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:01:14.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:385: _scrub_abort: break 2024-09-13T16:01:14.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:387: _scrub_abort: set +o pipefail 2024-09-13T16:01:14.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:389: _scrub_abort: sleep 5 2024-09-13T16:01:19.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:391: _scrub_abort: grep 'nodeep_scrub set, aborting' td/osd-scrub-test/osd.1.log 2024-09-13T16:01:19.183 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T15:59:12.901+0000 7fe27ee9a640 10 osd.1 pg_epoch: 21 pg[1.0( v 19'1000 (0'0,19'1000] local-lis/les=17/18 n=1000 ec=17/17 lis/c=17/17 les/c/f=18/18/0 sis=17) [1,0,2] r=0 lpr=17 crt=19'1000 lcod 19'999 mlcod 19'999 active+clean+scrubbing+deep [ 1.0: ] ] scrubber: nodeep_scrub set, aborting 2024-09-13T16:01:19.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:397: _scrub_abort: get_last_scrub_stamp 1.0 2024-09-13T16:01:19.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:19.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:19.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:19.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:19.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:397: _scrub_abort: local last_scrub=2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:19.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:398: _scrub_abort: ceph config set osd osd_scrub_sleep 0.1 2024-09-13T16:01:19.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:400: _scrub_abort: ceph osd unset nodeep-scrub 2024-09-13T16:01:20.409 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is unset 2024-09-13T16:01:20.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:401: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2024-09-13T16:01:20.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:403: _scrub_abort: ceph osd unset noscrub 2024-09-13T16:01:20.944 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is unset 2024-09-13T16:01:20.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:405: _scrub_abort: TIMEOUT=500 2024-09-13T16:01:20.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:406: _scrub_abort: wait_for_scrub 1.0 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:20.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T16:01:20.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:20.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T16:01:20.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T16:01:20.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:20.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:20.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:20.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:20.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:20.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:21.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:21.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:22.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:22.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:22.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:22.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:22.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:22.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:22.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:22.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:22.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:23.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:23.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:23.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:23.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:23.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:23.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:23.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:23.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:23.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:24.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:24.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:24.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:24.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:24.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:24.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:24.971 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:25.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:25.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:26.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:26.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:26.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:26.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:26.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:26.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:26.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:26.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:27.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:27.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:27.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:27.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:27.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:27.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:27.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:27.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:27.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:28.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:28.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:28.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:28.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:28.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:28.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:28.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:29.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:29.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:30.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:30.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:30.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:30.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:30.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:30.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:30.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:30.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:30.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:31.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:31.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:31.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:31.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:31.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:31.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:31.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:31.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:31.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:32.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:32.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:32.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:32.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:32.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:32.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:32.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:33.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:33.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:34.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:34.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:34.236 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:34.236 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:34.236 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:34.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:34.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:34.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:34.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:35.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:35.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:35.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:35.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:35.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:35.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:35.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:35.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:35.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:36.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:36.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:36.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:36.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:36.903 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:36.903 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:36.903 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:37.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:37.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:38.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:38.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:38.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:38.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:38.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:38.232 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:38.232 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:38.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:38.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:39.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:39.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:39.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:39.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:39.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:39.546 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:39.546 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:39.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:39.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:40.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:40.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:40.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:40.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:40.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:40.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:40.863 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:41.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:41.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:42.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:42.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:42.197 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:42.197 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:42.197 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:42.198 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:42.198 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:42.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:42.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:43.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:43.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:43.516 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:43.516 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:43.516 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:43.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:43.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:43.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:43.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:44.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:44.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:44.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:44.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:44.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:44.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:44.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:45.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:45.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:46.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:46.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:46.154 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:46.154 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:46.154 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:46.155 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:46.155 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:46.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:46.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:47.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:47.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:47.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:47.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:47.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:47.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:47.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:47.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:47.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:48.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:48.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:48.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:48.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:48.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:48.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:48.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:49.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:49.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:50.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:50.128 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:50.128 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:50.128 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:50.128 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:50.128 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:50.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:50.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:51.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:51.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:51.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:51.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:51.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:51.445 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:51.445 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:51.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:51.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:52.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:52.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:52.773 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:52.773 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:52.773 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:52.773 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:52.773 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:53.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:53.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:54.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:54.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:54.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:54.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:54.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:54.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:54.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:54.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:54.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:55.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:55.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:55.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:55.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:55.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:55.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:55.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:55.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:55.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:56.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:56.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:56.767 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:56.767 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:56.767 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:56.767 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:56.767 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:57.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:57.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:58.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:58.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:58.101 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:58.101 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:58.101 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:58.101 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:58.101 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:58.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:01:59.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:01:59.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:01:59.416 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:01:59.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:01:59.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:01:59.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:01:59.417 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:01:59.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:01:59.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:00.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:00.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:00.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:00.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:00.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:00.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:00.728 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:01.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:01.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:02.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:02.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:02.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:02.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:02.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:02.053 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:02.053 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:02.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:02.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:03.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:03.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:03.373 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:03.373 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:03.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:03.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:03.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:03.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:03.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:04.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:04.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:04.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:04.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:04.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:04.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:04.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:05.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:05.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:06.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:06.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:06.061 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:06.061 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:06.061 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:06.062 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:06.062 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:06.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:06.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:07.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:07.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:07.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:07.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:07.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:07.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:07.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:07.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:07.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:08.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:08.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:08.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:08.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:08.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:08.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:08.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:09.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:09.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:10.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:10.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:10.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:10.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:10.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:10.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:10.053 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:10.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:10.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:11.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:11.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:11.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:11.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:11.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:11.375 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:11.375 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:11.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:11.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:12.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:12.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:12.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:12.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:12.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:12.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:12.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:12.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:12.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:13.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:13.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:13.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:13.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:13.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:13.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:13.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:14.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:14.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:15.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:15.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:15.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:15.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:15.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:15.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:15.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:15.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:16.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:16.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:16.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:16.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:16.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:16.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:16.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:17.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:17.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:18.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:18.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:18.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:18.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:18.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:18.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:18.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:18.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:18.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:19.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:19.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:19.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:19.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:19.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:19.320 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:19.320 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:19.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:19.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:20.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:20.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:20.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:20.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:20.644 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:20.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:20.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:20.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:20.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:21.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:21.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:21.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:21.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:21.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:21.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:21.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:22.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:22.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:23.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:23.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:23.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:23.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:23.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:23.293 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:23.293 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:23.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:23.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:24.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:24.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:24.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:24.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:24.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:24.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:24.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:24.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:24.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:25.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:25.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:25.961 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:25.961 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:25.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:25.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:25.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:26.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:26.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:27.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:27.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:27.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:27.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:27.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:27.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:27.292 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:27.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:27.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:28.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:28.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:28.609 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:28.609 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:28.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:28.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:28.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:28.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:28.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:29.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:29.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:29.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:29.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:29.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:29.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:29.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:30.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:30.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:31.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:31.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:31.269 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:31.270 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:31.270 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:31.270 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:31.270 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:31.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:31.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:32.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:32.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:32.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:32.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:32.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:32.603 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:32.603 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:32.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:32.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:33.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:33.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:33.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:33.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:33.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:33.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:33.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:34.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:34.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:35.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:35.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:35.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:35.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:35.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:35.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:35.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:35.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:35.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:36.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:36.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:36.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:36.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:36.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:36.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:36.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:36.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:36.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:37.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:37.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:37.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:37.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:37.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:37.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:37.926 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:38.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:38.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:39.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:39.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:39.254 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:39.254 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:39.254 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:39.254 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:39.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:39.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:39.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:40.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:40.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:40.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:40.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:40.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:40.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:40.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:40.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:40.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:41.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:41.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:41.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:41.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:41.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:41.917 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:41.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:42.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:42.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:43.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:43.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:43.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:43.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:43.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:43.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:43.236 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:43.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:43.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:44.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:44.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:44.559 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:44.559 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:44.559 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:44.560 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:44.560 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:44.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:44.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:45.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:45.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:45.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:45.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:45.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:45.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:45.877 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:46.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:46.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:47.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:47.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:47.199 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:47.199 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:47.199 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:47.200 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:47.200 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:47.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:47.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:48.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:48.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:48.512 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:48.512 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:48.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:48.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:48.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:48.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:48.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:49.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:49.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:49.847 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:49.847 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:49.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:49.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:49.848 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:50.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:50.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:51.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:51.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:51.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:51.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:51.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:51.172 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:51.172 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:51.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:51.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:52.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:52.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:52.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:52.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:52.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:52.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:52.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:52.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:52.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:53.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:53.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:53.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:53.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:53.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:53.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:53.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:54.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:54.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:55.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:55.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:55.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:55.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:55.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:55.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:55.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:55.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:55.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:56.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:56.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:56.464 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:56.464 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:56.464 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:56.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:56.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:56.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:56.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:57.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:57.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:57.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:57.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:57.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:57.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:57.779 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:58.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:58.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:02:59.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:02:59.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:02:59.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:02:59.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:02:59.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:02:59.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:02:59.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:02:59.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:02:59.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:00.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:00.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:00.427 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:00.427 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:00.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:00.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:00.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:00.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:00.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:01.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:01.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:01.747 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:01.747 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:01.747 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:01.747 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:01.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:02.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:02.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:03.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:03.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:03.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:03.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:03.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:03.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:03.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:03.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:03.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:04.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:04.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:04.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:04.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:04.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:04.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:04.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:04.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:04.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:05.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:05.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:05.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:05.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:05.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:05.749 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:05.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:06.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:06.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:07.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:07.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:07.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:07.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:07.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:07.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:07.077 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:07.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:07.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:08.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:08.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:08.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:08.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:08.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:08.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:08.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:08.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:08.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:09.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:09.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:09.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:09.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:09.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:09.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:09.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:10.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:10.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:11.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:11.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:11.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:11.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:11.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:11.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:11.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:11.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:11.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:12.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:12.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:12.361 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:12.361 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:12.361 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:12.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:12.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:12.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:12.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:13.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:13.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:13.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:13.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:13.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:13.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:13.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:14.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:14.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:15.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:15.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:15.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:15.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:15.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:15.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:15.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:15.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:15.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:16.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:16.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:16.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:16.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:16.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:16.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:16.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:16.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:16.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:17.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:17.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:17.676 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:17.676 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:17.676 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:17.677 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:17.677 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:17.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:17.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:18.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:18.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:18.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:18.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:18.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:18.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:18.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:19.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:19.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:20.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:20.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:20.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:20.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:20.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:20.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:20.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:20.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:20.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:21.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:21.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:21.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:21.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:21.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:21.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:21.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:21.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:21.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:22.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:22.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:22.952 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:22.952 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:22.952 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:22.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:22.953 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:23.281 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:23.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:24.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:24.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:24.283 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:24.283 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:24.283 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:24.283 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:24.284 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:24.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:24.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:25.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:25.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:25.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:25.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:25.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:25.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:25.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:25.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:25.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:26.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:26.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:26.947 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:26.947 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:26.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:26.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:26.948 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:27.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:27.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:28.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:28.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:28.278 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:28.279 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:28.279 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:28.279 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:28.279 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:28.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:28.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:29.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:29.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:29.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:29.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:29.599 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:29.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:29.600 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:29.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:29.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:30.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:30.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:30.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:30.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:30.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:30.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:30.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:31.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:31.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:32.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:32.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:32.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:32.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:32.243 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:32.243 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:32.243 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:32.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:32.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:33.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:33.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:33.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:33.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:33.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:33.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:33.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:33.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:33.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:34.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:34.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:34.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:34.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:34.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:34.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:34.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:35.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:35.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:36.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:36.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:36.233 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:36.233 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:36.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:36.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:36.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:36.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:36.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:37.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:37.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:37.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:37.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:37.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:37.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:37.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:37.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:37.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:38.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:38.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:38.893 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:38.893 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:38.893 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:38.894 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:38.894 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:39.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:39.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:40.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:40.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:40.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:40.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:40.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:40.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:40.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:40.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:40.528 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:41.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:41.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:41.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:41.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:41.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:41.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:41.531 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:41.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:41.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:42.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:42.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:42.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:42.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:42.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:42.866 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:42.866 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:43.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:43.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:44.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:44.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:44.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:44.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:44.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:44.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:44.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:44.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:44.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:45.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:45.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:45.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:45.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:45.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:45.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:45.495 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:45.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:45.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:46.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:46.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:46.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:46.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:46.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:46.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:46.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:47.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:47.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:48.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:48.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:48.130 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:48.130 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:48.130 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:48.130 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:48.130 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:48.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:48.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:49.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:49.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:49.459 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:49.459 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:49.459 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:49.459 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:49.460 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:49.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:49.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:50.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:50.773 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:50.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:50.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:50.774 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:50.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:50.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:51.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:51.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:52.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:52.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:52.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:52.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:52.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:52.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:52.107 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:52.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:52.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:53.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:53.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:53.427 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:53.427 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:53.427 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:53.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:53.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:53.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:53.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:54.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:54.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:54.758 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:54.758 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:54.758 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:54.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:54.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:55.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:55.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:56.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:56.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:56.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:56.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:56.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:56.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:56.092 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:56.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:56.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:57.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:57.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:57.433 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:57.433 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:57.433 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:57.433 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:57.434 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:57.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:57.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:03:58.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:03:58.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:03:58.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:03:58.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:03:58.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:03:58.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:03:58.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:03:59.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:03:59.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:00.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:00.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:00.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:00.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:00.090 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:00.091 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:00.091 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:00.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:00.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:01.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:01.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:01.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:01.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:01.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:01.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:01.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:01.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:01.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:02.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:02.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:02.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:02.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:02.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:02.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:02.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:03.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:03.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:04.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:04.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:04.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:04.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:04.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:04.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:04.080 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:04.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:04.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:05.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:05.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:05.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:05.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:05.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:05.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:05.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:05.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:05.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:06.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:06.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:06.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:06.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:06.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:06.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:06.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:07.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:07.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:08.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:08.069 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:08.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:08.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:08.070 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:08.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:08.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:08.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:08.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:09.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:09.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:09.383 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:09.383 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:09.383 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:09.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:09.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:09.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:09.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:10.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:10.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:10.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:10.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:10.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:10.714 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:10.714 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:11.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:11.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:12.078 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:12.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:12.080 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:12.080 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:12.080 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:12.080 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:12.080 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:12.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:12.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:13.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:13.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:13.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:13.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:13.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:13.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:13.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:13.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:13.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:14.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:14.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:14.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:14.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:14.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:14.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:14.740 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:15.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:15.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:16.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:16.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:16.055 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:16.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:16.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:16.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:16.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:16.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:16.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:17.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:17.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:17.365 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:17.366 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:17.366 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:17.366 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:17.366 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:17.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:17.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:18.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:18.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:18.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:18.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:18.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:18.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:18.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:19.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:19.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:20.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:20.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:20.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:20.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:20.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:20.007 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:20.007 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:20.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:20.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:21.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:21.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:21.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:21.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:21.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:21.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:21.333 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:21.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:21.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:22.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:22.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:22.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:22.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:22.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:22.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:22.663 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:22.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:22.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:23.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:23.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:23.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:23.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:23.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:23.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:23.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:24.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:24.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:25.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:25.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:25.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:25.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:25.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:25.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:25.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:25.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:25.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:26.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:26.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:26.651 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:26.651 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:26.651 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:26.651 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:26.651 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:26.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:26.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:27.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:27.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:27.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:27.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:27.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:27.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:27.976 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:28.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:28.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:29.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:29.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:29.305 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:29.305 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:29.305 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:29.306 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:29.306 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:29.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:29.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:30.620 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:30.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:30.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:30.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:30.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:30.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:30.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:31.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:31.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:32.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:32.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:32.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:32.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:32.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:32.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:32.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:32.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:32.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:33.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:33.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:33.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:33.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:33.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:33.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:33.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:33.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:33.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:34.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:34.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:34.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:34.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:34.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:34.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:34.683 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:35.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:35.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:36.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:36.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:36.051 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:36.051 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:36.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:36.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:36.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:36.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:36.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:37.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:37.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:37.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:37.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:37.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:37.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:37.386 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:37.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:37.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:38.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:38.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:38.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:38.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:38.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:38.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:38.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:39.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:39.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:40.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:40.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:40.046 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:40.046 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:40.046 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:40.046 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:40.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:40.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:40.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:41.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:41.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:41.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:41.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:41.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:41.381 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:41.381 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:41.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:41.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:42.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:42.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:42.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:42.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:42.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:42.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:42.711 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:43.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:43.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:44.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:44.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:44.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:44.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:44.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:44.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:44.042 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:44.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:44.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:45.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:45.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:45.364 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:45.364 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:45.365 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:45.365 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:45.365 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:45.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:45.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:46.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:46.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:46.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:46.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:46.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:46.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:46.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:47.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:47.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:48.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:48.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:48.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:48.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:48.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:48.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:48.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:48.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:48.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:49.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:49.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:49.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:49.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:49.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:49.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:49.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:49.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:49.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:50.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:50.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:50.735 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:50.735 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:50.735 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:50.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:50.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:51.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:51.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:52.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:52.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:52.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:52.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:52.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:52.068 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:52.068 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:52.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-08-30T15:57:27.504191+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:52.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:04:53.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:04:53.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:04:53.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:04:53.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:04:53.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:04:53.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:04:53.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:04:53.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:04:51.072426+0000 '>' 2024-08-30T15:57:27.504191+0000 2024-09-13T16:04:53.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T16:04:53.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:407: _scrub_abort: perf_counters td/osd-scrub-test 3 2024-09-13T16:04:53.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-09-13T16:04:53.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2024-09-13T16:04:53.728 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2024-09-13T16:04:53.729 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2024-09-13T16:04:53.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:04:53.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-09-13T16:04:53.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:04:53.851 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:04:53.851 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:04:53.851 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:04:53.851 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:04:53.851 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:04:53.851 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:04:53.851 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.851 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:04:53.851 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:04:53.852 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:04:53.852 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:04:53.852 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:04:53.852 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.852 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.852 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.852 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.852 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:04:53.852 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:04:53.852 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.852 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.852 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.852 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.853 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:04:53.853 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:04:53.853 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:04:53.853 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:04:53.853 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:04:53.853 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:04:53.853 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:04:53.853 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.853 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.854 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.854 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.854 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:04:53.854 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:04:53.854 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:04:53.854 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:04:53.854 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.854 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.854 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.854 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.854 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:04:53.854 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:53.854 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:53.854 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:04:53.855 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:04:53.855 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:04:53.855 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:04:53.855 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:04:53.855 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:04:53.855 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.855 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:04:53.855 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:04:53.855 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:04:53.855 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:04:53.855 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:04:53.856 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.856 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.856 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.856 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.856 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:04:53.856 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:04:53.856 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.856 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.856 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.856 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.857 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:04:53.857 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:04:53.857 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:04:53.857 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:04:53.857 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:04:53.857 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:04:53.857 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:04:53.857 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.857 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.857 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.857 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.857 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:04:53.857 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:04:53.858 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:04:53.858 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:04:53.858 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.858 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.858 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.858 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.858 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:04:53.858 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:53.858 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:53.858 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:04:53.858 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:04:53.858 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:04:53.858 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:04:53.858 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:04:53.859 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:04:53.859 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.859 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:04:53.859 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:04:53.859 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:04:53.859 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:04:53.859 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:04:53.859 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.859 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.859 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.859 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.859 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:04:53.859 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:04:53.859 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.860 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.860 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.860 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.860 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:04:53.860 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:04:53.860 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:04:53.860 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:04:53.860 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:04:53.860 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:04:53.860 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:04:53.860 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.860 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.860 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.861 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.861 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:04:53.861 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:04:53.861 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:04:53.861 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:04:53.861 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.861 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.861 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.861 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.861 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:04:53.861 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:53.861 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:53.861 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:04:53.861 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:04:53.862 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:04:53.862 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:04:53.862 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:04:53.862 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:04:53.862 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.862 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:04:53.862 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:04:53.862 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:04:53.862 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:04:53.862 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:04:53.862 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.862 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.862 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.863 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.863 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:04:53.863 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:04:53.863 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.863 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.863 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.863 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.863 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:04:53.863 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:04:53.863 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:04:53.864 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:04:53.864 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:04:53.864 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:04:53.864 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:04:53.864 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.864 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.865 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.865 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.865 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:04:53.865 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:04:53.865 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:04:53.865 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:04:53.865 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.865 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.865 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.865 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.866 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:04:53.866 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:53.866 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:53.866 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:04:53.866 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:04:53.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:04:53.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-09-13T16:04:53.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:04:53.994 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:04:53.994 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:04:53.994 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:04:53.995 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:04:53.995 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:04:53.995 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:04:53.995 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.995 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:04:53.995 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:04:53.996 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:04:53.996 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:04:53.996 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:04:53.996 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.996 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.997 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.997 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.997 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:04:53.997 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:04:53.997 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.997 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.998 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.998 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.998 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:04:53.998 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:04:53.998 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:04:53.998 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:04:53.998 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:04:53.998 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:04:53.998 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:04:53.998 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.998 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.998 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.998 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.998 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:04:53.999 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:04:53.999 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:04:53.999 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:04:53.999 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:53.999 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:53.999 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:53.999 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:53.999 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:04:53.999 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:53.999 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:53.999 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:04:54.000 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:04:54.000 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:04:54.000 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:04:54.000 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:04:54.000 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:04:54.000 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.000 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:04:54.000 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 2, 2024-09-13T16:04:54.000 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 2, 2024-09-13T16:04:54.000 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 1, 2024-09-13T16:04:54.000 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:04:54.000 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 1, 2024-09-13T16:04:54.001 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 209.166517784, 2024-09-13T16:04:54.001 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 209.166517784 2024-09-13T16:04:54.001 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.001 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 1, 2024-09-13T16:04:54.001 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:04:54.001 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 1, 2024-09-13T16:04:54.001 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 4.999867509, 2024-09-13T16:04:54.001 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 4.999867509 2024-09-13T16:04:54.001 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.001 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:04:54.001 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 41, 2024-09-13T16:04:54.002 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:04:54.002 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:04:54.002 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:04:54.002 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 2, 2024-09-13T16:04:54.002 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:04:54.002 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 2, 2024-09-13T16:04:54.002 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0.151995998, 2024-09-13T16:04:54.002 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0.075997999 2024-09-13T16:04:54.002 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.002 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:04:54.002 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:04:54.002 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:04:54.002 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:04:54.002 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.003 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.003 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.003 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.003 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 2 2024-09-13T16:04:54.003 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:54.003 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:54.003 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:04:54.003 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:04:54.003 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:04:54.003 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:04:54.003 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:04:54.004 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:04:54.004 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.004 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:04:54.004 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:04:54.004 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:04:54.004 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:04:54.004 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:04:54.004 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.004 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.004 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.004 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.004 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:04:54.004 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:04:54.005 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.005 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.005 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.005 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.005 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:04:54.005 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:04:54.005 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:04:54.005 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:04:54.005 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:04:54.005 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:04:54.005 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:04:54.005 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.005 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.005 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.006 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.006 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:04:54.006 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:04:54.006 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:04:54.006 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:04:54.006 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.006 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.006 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.006 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.006 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:04:54.006 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:54.006 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:54.006 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:04:54.006 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:04:54.007 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:04:54.007 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:04:54.007 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:04:54.007 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:04:54.007 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.007 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:04:54.007 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:04:54.007 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:04:54.007 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:04:54.007 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:04:54.007 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.007 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.007 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.008 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.008 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:04:54.008 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:04:54.008 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.008 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.008 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.008 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.008 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:04:54.008 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:04:54.008 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:04:54.008 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:04:54.008 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:04:54.008 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:04:54.009 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:04:54.009 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.009 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.009 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.009 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.009 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:04:54.009 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:04:54.009 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:04:54.009 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:04:54.010 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.010 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.010 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.010 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.010 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:04:54.010 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:54.010 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:54.010 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:04:54.011 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:04:54.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:04:54.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2024-09-13T16:04:54.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:04:54.122 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:04:54.122 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:04:54.123 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:04:54.123 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:04:54.123 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:04:54.123 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:04:54.123 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.123 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:04:54.123 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:04:54.123 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:04:54.123 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:04:54.123 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:04:54.123 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.123 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.123 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.124 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.124 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:04:54.124 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:04:54.124 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.124 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.124 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.124 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.124 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:04:54.124 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:04:54.124 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:04:54.124 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:04:54.124 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:04:54.124 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:04:54.125 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:04:54.125 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.125 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.125 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.125 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.125 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:04:54.125 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:04:54.125 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:04:54.125 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:04:54.125 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.125 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.125 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.125 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.125 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:04:54.126 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:54.126 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:54.126 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:04:54.126 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:04:54.126 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:04:54.126 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:04:54.126 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:04:54.126 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:04:54.126 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.126 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:04:54.127 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:04:54.127 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:04:54.127 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:04:54.127 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:04:54.127 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.127 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.127 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.127 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.127 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:04:54.127 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:04:54.127 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.127 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.127 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.128 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.128 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:04:54.128 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:04:54.128 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:04:54.128 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:04:54.128 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:04:54.128 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:04:54.128 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:04:54.128 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.128 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.128 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.128 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.128 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:04:54.128 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:04:54.129 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:04:54.129 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:04:54.129 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.129 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.129 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.129 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.129 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:04:54.129 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:54.129 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:54.129 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:04:54.129 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:04:54.129 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:04:54.129 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:04:54.130 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:04:54.130 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:04:54.130 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.130 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:04:54.130 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:04:54.130 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:04:54.130 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:04:54.130 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:04:54.130 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.130 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.130 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.130 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.130 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:04:54.131 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:04:54.131 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.131 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.131 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.131 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.131 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:04:54.131 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:04:54.131 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:04:54.131 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:04:54.131 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:04:54.131 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:04:54.131 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:04:54.131 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.132 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.132 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.132 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.132 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:04:54.132 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:04:54.132 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:04:54.132 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:04:54.132 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.132 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.132 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.132 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.132 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:04:54.132 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:54.133 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:54.133 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:04:54.133 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:04:54.133 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:04:54.133 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:04:54.133 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:04:54.133 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:04:54.133 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.133 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:04:54.133 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:04:54.133 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:04:54.133 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:04:54.133 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:04:54.133 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.134 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.134 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.134 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.134 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:04:54.134 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:04:54.134 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.134 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.134 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.134 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.134 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:04:54.134 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:04:54.134 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:04:54.134 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:04:54.134 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:04:54.135 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:04:54.135 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:04:54.135 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.135 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.135 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.135 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.135 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:04:54.135 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:04:54.136 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:04:54.136 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:04:54.136 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:04:54.136 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:04:54.136 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:04:54.136 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:04:54.136 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:04:54.136 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:54.136 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:04:54.136 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:04:54.136 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:04:54.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_deep_scrub_abort ------------------' 2024-09-13T16:04:54.137 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Teardown Test TEST_deep_scrub_abort ------------------ 2024-09-13T16:04:54.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-09-13T16:04:54.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:04:54.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:04:54.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:04:54.138 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:04:54.138 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:04:54.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:04:54.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:04:54.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:04:54.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:04:54.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:04:54.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:04:54.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:04:54.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:04:54.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:04:54.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:04:54.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:04:54.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:04:54.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:04:54.247 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:04:54.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:04:54.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:04:54.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:04:54.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:04:54.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:04:54.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:04:54.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:04:54.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:04:54.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:04:54.301 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Complete Test TEST_deep_scrub_abort ------------------ 2024-09-13T16:04:54.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_deep_scrub_abort ------------------' 2024-09-13T16:04:54.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-13T16:04:54.301 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Prepare Test TEST_dump_scrub_schedule ------------------- 2024-09-13T16:04:54.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_dump_scrub_schedule -------------------' 2024-09-13T16:04:54.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-09-13T16:04:54.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2024-09-13T16:04:54.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2024-09-13T16:04:54.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:04:54.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:04:54.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:04:54.303 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:04:54.303 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:04:54.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:04:54.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:04:54.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:04:54.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:04:54.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:04:54.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:04:54.304 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:04:54.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:04:54.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:04:54.305 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:04:54.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:04:54.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:04:54.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:04:54.306 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:04:54.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:04:54.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:04:54.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:04:54.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:04:54.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:04:54.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:04:54.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:04:54.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:04:54.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:04:54.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2024-09-13T16:04:54.311 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T16:04:54.311 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:04:54.311 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:04:54.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.528094 2024-09-13T16:04:54.312 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T16:04:54.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T16:04:54.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T16:04:54.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-13T16:04:54.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_dump_scrub_schedule -----------------------' 2024-09-13T16:04:54.313 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Run Test TEST_dump_scrub_schedule ----------------------- 2024-09-13T16:04:54.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_dump_scrub_schedule td/osd-scrub-test 2024-09-13T16:04:54.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:528: TEST_dump_scrub_schedule: local dir=td/osd-scrub-test 2024-09-13T16:04:54.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:529: TEST_dump_scrub_schedule: local poolname=test 2024-09-13T16:04:54.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:530: TEST_dump_scrub_schedule: local OSDS=3 2024-09-13T16:04:54.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:531: TEST_dump_scrub_schedule: local objects=15 2024-09-13T16:04:54.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:533: TEST_dump_scrub_schedule: TESTDATA=testdata.528094 2024-09-13T16:04:54.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-13T16:04:54.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2024-09-13T16:04:54.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T16:04:54.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T16:04:54.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T16:04:54.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2024-09-13T16:04:54.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-13T16:04:54.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T16:04:54.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:04:54.826 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:04:54.827 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:04:54.827 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:04:54.827 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:04:54.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:04:54.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-13T16:04:55.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T16:04:55.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T16:04:55.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T16:04:55.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T16:04:55.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T16:04:55.055 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T16:04:55.055 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T16:04:55.055 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T16:04:55.055 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T16:04:55.055 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:04:55.055 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:04:55.055 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:04:55.056 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-mon.a.asok 2024-09-13T16:04:55.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T16:04:55.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.528094/ceph-mon.a.asok config get fsid 2024-09-13T16:04:55.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T16:04:55.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T16:04:55.173 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T16:04:55.174 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T16:04:55.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T16:04:55.175 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T16:04:55.175 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T16:04:55.175 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T16:04:55.175 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:04:55.175 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:04:55.175 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:04:55.175 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-mon.a.asok 2024-09-13T16:04:55.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T16:04:55.176 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.528094/ceph-mon.a.asok config get mon_host 2024-09-13T16:04:55.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:536: TEST_dump_scrub_schedule: run_mgr td/osd-scrub-test x 2024-09-13T16:04:55.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2024-09-13T16:04:55.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T16:04:55.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T16:04:55.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T16:04:55.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2024-09-13T16:04:55.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T16:04:56.228 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T16:04:56.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:04:56.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:04:56.229 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:04:56.229 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:04:56.230 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:04:56.230 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:04:56.230 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T16:04:56.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T16:04:56.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2' 2024-09-13T16:04:56.267 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: expr 3 - 1 2024-09-13T16:04:56.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: seq 0 2 2024-09-13T16:04:56.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:04:56.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:551: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 0 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-13T16:04:56.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:04:56.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:04:56.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T16:04:56.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:04:56.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-13T16:04:56.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:04:56.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:04:56.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:04:56.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:04:56.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-13T16:04:56.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-13T16:04:56.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:04:56.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:04:56.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:04:56.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:04:56.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:04:56.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:04:56.272 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:04:56.272 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:04:56.272 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:04:56.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:04:56.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:04:56.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:04:56.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:04:56.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:04:56.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:04:56.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:04:56.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:04:56.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:04:56.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:04:56.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:04:56.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:04:56.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2' 2024-09-13T16:04:56.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-13T16:04:56.274 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:04:56.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=22eab50d-812a-4905-b5c2-145e51d45eb4 2024-09-13T16:04:56.274 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 22eab50d-812a-4905-b5c2-145e51d45eb4 2024-09-13T16:04:56.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 22eab50d-812a-4905-b5c2-145e51d45eb4' 2024-09-13T16:04:56.275 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:04:56.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCoYuRmsV9HERAAORKINlugl45lybV8dB6f/A== 2024-09-13T16:04:56.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCoYuRmsV9HERAAORKINlugl45lybV8dB6f/A=="}' 2024-09-13T16:04:56.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 22eab50d-812a-4905-b5c2-145e51d45eb4 -i td/osd-scrub-test/0/new.json 2024-09-13T16:04:56.598 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:04:56.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-13T16:04:56.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 --mkfs --key AQCoYuRmsV9HERAAORKINlugl45lybV8dB6f/A== --osd-uuid 22eab50d-812a-4905-b5c2-145e51d45eb4 2024-09-13T16:04:56.660 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:04:56.660+0000 7f23f2f257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:04:56.681 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:04:56.680+0000 7f23f2f257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:04:56.683 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:04:56.684+0000 7f23f2f257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:04:56.683 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:04:56.684+0000 7f23f2f257c0 -1 bdev(0x55c2d94e9c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2024-09-13T16:04:56.684 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:04:56.684+0000 7f23f2f257c0 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-13T16:05:04.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-13T16:05:04.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:05:04.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T16:05:04.818 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T16:05:04.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:05:05.226 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T16:05:05.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T16:05:05.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-13T16:05:05.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:05:05.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:05:05.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:05:05.262 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:05.259+0000 7f11689b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:05:05.306 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:05.303+0000 7f11689b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:05:05.307 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:05.307+0000 7f11689b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:05:05.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T16:05:05.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:05:05.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T16:05:05.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:05:05.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:05:05.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:05.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:05:05.554 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:05:05.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:05.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:05:05.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:05:06.834 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:06.831+0000 7f11689b47c0 -1 Falling back to public interface 2024-09-13T16:05:06.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:05:06.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:06.850 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:05:06.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:05:06.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:06.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:05:07.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:05:08.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:05:08.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:08.181 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:05:08.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:05:08.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:08.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:05:08.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:05:09.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:05:09.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:09.507 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:05:09.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:05:09.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:09.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:05:09.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:05:10.246 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:10.243+0000 7f11689b47c0 -1 osd.0 0 log_to_monitors true 2024-09-13T16:05:10.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:05:10.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:10.829 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:05:10.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:05:10.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:10.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:05:11.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:05:12.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:05:12.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:12.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:05:12.209 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:05:12.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:12.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:05:12.631 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2221003908,v1:127.0.0.1:6803/2221003908] [v2:127.0.0.1:6804/2221003908,v1:127.0.0.1:6805/2221003908] exists,up 22eab50d-812a-4905-b5c2-145e51d45eb4 2024-09-13T16:05:12.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:05:12.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:05:12.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:05:12.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:05:12.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:551: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 1 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-13T16:05:12.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:05:12.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:05:12.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T16:05:12.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:05:12.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-13T16:05:12.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:05:12.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:05:12.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:05:12.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:05:12.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-13T16:05:12.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-13T16:05:12.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:05:12.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:05:12.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:05:12.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:05:12.635 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:05:12.636 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:05:12.636 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:05:12.636 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:05:12.636 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:05:12.636 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:05:12.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:05:12.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:05:12.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:05:12.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:05:12.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:05:12.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:05:12.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:05:12.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:05:12.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:05:12.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:05:12.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:05:12.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2' 2024-09-13T16:05:12.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-13T16:05:12.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:05:12.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=76d9c8ae-d32a-4b8f-b895-3787a6c36a7e 2024-09-13T16:05:12.638 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 76d9c8ae-d32a-4b8f-b895-3787a6c36a7e 2024-09-13T16:05:12.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 76d9c8ae-d32a-4b8f-b895-3787a6c36a7e' 2024-09-13T16:05:12.638 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:05:12.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC4YuRmIM1MJxAAjs2938697Dw69BM6X0If5Q== 2024-09-13T16:05:12.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC4YuRmIM1MJxAAjs2938697Dw69BM6X0If5Q=="}' 2024-09-13T16:05:12.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 76d9c8ae-d32a-4b8f-b895-3787a6c36a7e -i td/osd-scrub-test/1/new.json 2024-09-13T16:05:13.143 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:05:13.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-13T16:05:13.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 --mkfs --key AQC4YuRmIM1MJxAAjs2938697Dw69BM6X0If5Q== --osd-uuid 76d9c8ae-d32a-4b8f-b895-3787a6c36a7e 2024-09-13T16:05:13.205 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:13.203+0000 7f2ff6b4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:05:13.287 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:13.283+0000 7f2ff6b4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:05:13.288 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:13.287+0000 7f2ff6b4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:05:13.288 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:13.287+0000 7f2ff6b4b7c0 -1 bdev(0x55de4f3ffc00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2024-09-13T16:05:13.288 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:13.287+0000 7f2ff6b4b7c0 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-13T16:05:21.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-13T16:05:21.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:05:21.773 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T16:05:21.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T16:05:21.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:05:22.193 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T16:05:22.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T16:05:22.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-13T16:05:22.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:05:22.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:05:22.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:05:22.231 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:22.231+0000 7ff252c807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:05:22.266 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:22.263+0000 7ff252c807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:05:22.268 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:22.267+0000 7ff252c807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:05:22.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T16:05:22.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:05:22.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T16:05:22.506 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:05:22.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:05:22.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:05:22.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:22.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:05:22.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:22.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:05:22.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:05:23.390 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:23.387+0000 7ff252c807c0 -1 Falling back to public interface 2024-09-13T16:05:23.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:05:23.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:23.843 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:05:23.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:05:23.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:23.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:05:24.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:05:25.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:05:25.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:25.160 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:05:25.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:05:25.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:25.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:05:25.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:05:26.449 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:26.447+0000 7ff252c807c0 -1 osd.1 0 log_to_monitors true 2024-09-13T16:05:26.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:05:26.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:26.487 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:05:26.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:05:26.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:26.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:05:26.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:05:27.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:05:27.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:27.798 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:05:27.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:05:27.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:27.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:05:28.112 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/389799681,v1:127.0.0.1:6811/389799681] [v2:127.0.0.1:6812/389799681,v1:127.0.0.1:6813/389799681] exists,up 76d9c8ae-d32a-4b8f-b895-3787a6c36a7e 2024-09-13T16:05:28.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:05:28.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:05:28.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:05:28.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:05:28.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:551: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 2 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-13T16:05:28.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:05:28.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:05:28.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T16:05:28.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:05:28.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-13T16:05:28.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:05:28.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:05:28.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:05:28.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:05:28.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-13T16:05:28.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-13T16:05:28.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:05:28.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:05:28.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:05:28.117 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:05:28.117 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:05:28.117 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:05:28.117 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:05:28.118 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:05:28.118 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:05:28.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:05:28.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:05:28.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:05:28.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:05:28.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:05:28.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:05:28.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:05:28.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:05:28.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:05:28.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:05:28.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:05:28.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:05:28.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2' 2024-09-13T16:05:28.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-13T16:05:28.119 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:05:28.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a6f7f843-6e28-46fa-b3a6-b5025483259a 2024-09-13T16:05:28.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 a6f7f843-6e28-46fa-b3a6-b5025483259a' 2024-09-13T16:05:28.120 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 a6f7f843-6e28-46fa-b3a6-b5025483259a 2024-09-13T16:05:28.120 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:05:28.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDIYuRmHlBXCBAA9kFSWWM6sYx/wEFfB5Q9Pg== 2024-09-13T16:05:28.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDIYuRmHlBXCBAA9kFSWWM6sYx/wEFfB5Q9Pg=="}' 2024-09-13T16:05:28.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a6f7f843-6e28-46fa-b3a6-b5025483259a -i td/osd-scrub-test/2/new.json 2024-09-13T16:05:28.570 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:05:28.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-13T16:05:28.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 --mkfs --key AQDIYuRmHlBXCBAA9kFSWWM6sYx/wEFfB5Q9Pg== --osd-uuid a6f7f843-6e28-46fa-b3a6-b5025483259a 2024-09-13T16:05:28.629 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:28.627+0000 7efcaaec87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:05:28.705 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:28.703+0000 7efcaaec87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:05:28.707 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:28.707+0000 7efcaaec87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:05:28.707 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:28.707+0000 7efcaaec87c0 -1 bdev(0x5642fb9a9c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2024-09-13T16:05:28.707 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:28.707+0000 7efcaaec87c0 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-13T16:05:37.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-13T16:05:37.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:05:37.191 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T16:05:37.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T16:05:37.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:05:37.598 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T16:05:37.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T16:05:37.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_sleep=0.2 2024-09-13T16:05:37.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:05:37.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:05:37.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:05:37.634 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:37.630+0000 7f2a474857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:05:37.678 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:37.674+0000 7f2a474857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:05:37.680 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:37.678+0000 7f2a474857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:05:37.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T16:05:37.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:05:37.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T16:05:37.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:05:37.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:05:37.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:37.907 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:05:37.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:05:37.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:37.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:05:38.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:05:38.982 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:38.978+0000 7f2a474857c0 -1 Falling back to public interface 2024-09-13T16:05:39.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:05:39.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:39.217 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:05:39.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:05:39.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:39.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:05:39.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:05:40.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:05:40.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:40.570 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:05:40.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:05:40.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:40.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:05:40.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:05:41.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:05:41.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:41.892 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:05:41.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:05:41.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:41.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:05:42.021 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:05:42.018+0000 7f2a474857c0 -1 osd.2 0 log_to_monitors true 2024-09-13T16:05:42.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:05:43.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:05:43.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:05:43.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:05:43.340 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:05:43.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:05:43.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:05:43.674 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1049903172,v1:127.0.0.1:6819/1049903172] [v2:127.0.0.1:6820/1049903172,v1:127.0.0.1:6821/1049903172] exists,up a6f7f843-6e28-46fa-b3a6-b5025483259a 2024-09-13T16:05:43.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:05:43.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:05:43.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:05:43.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: create_pool test 1 1 2024-09-13T16:05:43.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-13T16:05:44.125 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'test' created 2024-09-13T16:05:44.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T16:05:45.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:556: TEST_dump_scrub_schedule: wait_for_clean 2024-09-13T16:05:45.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:05:45.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:05:45.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:05:45.151 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:05:45.151 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:05:45.151 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:05:45.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:05:45.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:05:45.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:05:45.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:05:45.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:05:45.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:05:45.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:05:45.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:05:45.275 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:05:45.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:05:45.651 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:05:45.651 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:05:45.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:05:45.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:05:45.652 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:05:45.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836488 2024-09-13T16:05:45.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836488 2024-09-13T16:05:45.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488' 2024-09-13T16:05:45.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:05:45.797 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:05:46.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-13T16:05:46.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-13T16:05:46.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488 1-42949672965' 2024-09-13T16:05:46.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:05:46.019 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:05:46.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=60129542146 2024-09-13T16:05:46.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 60129542146 2024-09-13T16:05:46.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488 1-42949672965 2-60129542146' 2024-09-13T16:05:46.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:05:46.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836488 2024-09-13T16:05:46.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:05:46.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:05:46.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836488 2024-09-13T16:05:46.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:05:46.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836488 2024-09-13T16:05:46.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-13T16:05:46.215 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836488 2024-09-13T16:05:46.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:05:46.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836487 -lt 21474836488 2024-09-13T16:05:46.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:05:47.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:05:47.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:05:47.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836488 2024-09-13T16:05:47.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:05:47.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-13T16:05:47.851 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:05:47.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:05:47.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:05:47.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-13T16:05:47.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-13T16:05:47.856 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672965 2024-09-13T16:05:47.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-13T16:05:47.856 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:05:48.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-13T16:05:48.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:05:48.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-60129542146 2024-09-13T16:05:48.180 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:05:48.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:05:48.181 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-60129542146 2024-09-13T16:05:48.181 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:05:48.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=60129542146 2024-09-13T16:05:48.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2024-09-13T16:05:48.183 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 60129542146 2024-09-13T16:05:48.183 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:05:48.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 60129542146 -lt 60129542146 2024-09-13T16:05:48.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:05:48.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:05:48.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:05:48.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:05:48.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:05:48.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:05:48.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:05:48.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:05:48.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:05:48.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:05:48.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:05:49.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:05:49.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:05:49.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:05:49.339 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:05:49.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:05:49.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:05:49.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:05:49.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:557: TEST_dump_scrub_schedule: grep '^pool.*['\'']test['\'']' 2024-09-13T16:05:49.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:557: TEST_dump_scrub_schedule: ceph osd dump 2024-09-13T16:05:49.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:557: TEST_dump_scrub_schedule: awk '{ print $2 }' 2024-09-13T16:05:50.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:557: TEST_dump_scrub_schedule: poolid=1 2024-09-13T16:05:50.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:559: TEST_dump_scrub_schedule: dd if=/dev/urandom of=testdata.528094 bs=1032 count=1 2024-09-13T16:05:50.137 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T16:05:50.137 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T16:05:50.137 INFO:tasks.workunit.client.0.smithi112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.0001158 s, 8.9 MB/s 2024-09-13T16:05:50.138 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: seq 1 15 2024-09-13T16:05:50.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-13T16:05:50.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj1 testdata.528094 2024-09-13T16:05:50.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-13T16:05:50.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj2 testdata.528094 2024-09-13T16:05:50.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-13T16:05:50.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj3 testdata.528094 2024-09-13T16:05:50.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-13T16:05:50.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj4 testdata.528094 2024-09-13T16:05:50.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-13T16:05:50.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj5 testdata.528094 2024-09-13T16:05:51.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-13T16:05:51.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj6 testdata.528094 2024-09-13T16:05:51.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-13T16:05:51.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj7 testdata.528094 2024-09-13T16:05:51.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-13T16:05:51.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj8 testdata.528094 2024-09-13T16:05:51.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-13T16:05:51.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj9 testdata.528094 2024-09-13T16:05:51.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-13T16:05:51.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj10 testdata.528094 2024-09-13T16:05:52.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-13T16:05:52.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj11 testdata.528094 2024-09-13T16:05:52.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-13T16:05:52.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj12 testdata.528094 2024-09-13T16:05:52.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-13T16:05:52.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj13 testdata.528094 2024-09-13T16:05:52.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-13T16:05:52.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj14 testdata.528094 2024-09-13T16:05:52.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-09-13T16:05:52.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rados -p test put obj15 testdata.528094 2024-09-13T16:05:53.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:564: TEST_dump_scrub_schedule: rm -f testdata.528094 2024-09-13T16:05:53.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:566: TEST_dump_scrub_schedule: local pgid=1.0 2024-09-13T16:05:53.159 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:568: TEST_dump_scrub_schedule: date +%Y-%m-%dT%H:%M:%S.%N%:z 2024-09-13T16:05:53.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:568: TEST_dump_scrub_schedule: local now_is=2024-09-13T16:05:53.161849696+00:00 2024-09-13T16:05:53.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:576: TEST_dump_scrub_schedule: expct_starting=(['query_active']='false' ['query_is_future']='true' ['query_schedule']='scrub scheduled') 2024-09-13T16:05:53.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:576: TEST_dump_scrub_schedule: declare -A expct_starting 2024-09-13T16:05:53.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:577: TEST_dump_scrub_schedule: declare -A sched_data 2024-09-13T16:05:53.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:578: TEST_dump_scrub_schedule: extract_published_sch 1.0 2024-09-13T16:05:53.161849696+00:00 2019-10-12T20:32:43.645168+0000 sched_data 2024-09-13T16:05:53.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.0 2024-09-13T16:05:53.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2024-09-13T16:05:53.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-09-13T16:05:53.161849696+00:00 2024-09-13T16:05:53.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2019-10-12T20:32:43.645168+0000 2024-09-13T16:05:53.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2024-09-13T16:05:53.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-09-13T16:05:53.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-09-13T16:05:53.463 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:05:53.479 INFO:tasks.workunit.client.0.smithi112.stdout:from pg dump pg: { 2024-09-13T16:05:53.480 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_pg_state": "active+clean", 2024-09-13T16:05:53.480 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_state_has_scrubbing": false, 2024-09-13T16:05:53.480 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_last_duration": 0, 2024-09-13T16:05:53.480 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-13T16:05:53.480 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule_at": "2024-09-14T16:05:44.046351+0000", 2024-09-13T16:05:53.480 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_is_future": true, 2024-09-13T16:05:53.480 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_vs_date": true, 2024-09-13T16:05:53.480 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_reported_epoch": 17, 2024-09-13T16:05:53.480 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_seq": 15 2024-09-13T16:05:53.480 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:05:53.480 INFO:tasks.workunit.client.0.smithi112.stdout:query output: 2024-09-13T16:05:53.625 INFO:tasks.workunit.client.0.smithi112.stdout: "scrubber": { 2024-09-13T16:05:53.625 INFO:tasks.workunit.client.0.smithi112.stdout: "active": false, 2024-09-13T16:05:53.625 INFO:tasks.workunit.client.0.smithi112.stdout: "must_scrub": false, 2024-09-13T16:05:53.625 INFO:tasks.workunit.client.0.smithi112.stdout: "must_deep_scrub": false, 2024-09-13T16:05:53.625 INFO:tasks.workunit.client.0.smithi112.stdout: "must_repair": false, 2024-09-13T16:05:53.625 INFO:tasks.workunit.client.0.smithi112.stdout: "need_auto": false, 2024-09-13T16:05:53.625 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reg_stamp": "2024-09-14T16:05:44.046351+0000", 2024-09-13T16:05:53.625 INFO:tasks.workunit.client.0.smithi112.stdout: "schedule": "scrub scheduled @ 2024-09-14T16:05:44.046 (2024-09-14T16:05:44.046)" 2024-09-13T16:05:53.625 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:05:53.625 INFO:tasks.workunit.client.0.smithi112.stdout: "agent_state": {} 2024-09-13T16:05:53.795 INFO:tasks.workunit.client.0.smithi112.stdout:( 2024-09-13T16:05:53.795 INFO:tasks.workunit.client.0.smithi112.stdout:[query_epoch]=17 2024-09-13T16:05:53.795 INFO:tasks.workunit.client.0.smithi112.stdout:[query_seq]=32 2024-09-13T16:05:53.795 INFO:tasks.workunit.client.0.smithi112.stdout:[query_active]=false 2024-09-13T16:05:53.795 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule]='scrub scheduled' 2024-09-13T16:05:53.796 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule_at]='2024-09-14T16:05:44.046' 2024-09-13T16:05:53.796 INFO:tasks.workunit.client.0.smithi112.stdout:[query_target_at]='2024-09-14T16:05:44.046' 2024-09-13T16:05:53.796 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_duration]=0 2024-09-13T16:05:53.796 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_stamp]='2024-09-13T16:05:44.046351+0000' 2024-09-13T16:05:53.796 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_scrub]='0x0' 2024-09-13T16:05:53.796 INFO:tasks.workunit.client.0.smithi112.stdout:[query_is_future]=true 2024-09-13T16:05:53.796 INFO:tasks.workunit.client.0.smithi112.stdout:[query_vs_date]=true 2024-09-13T16:05:53.796 INFO:tasks.workunit.client.0.smithi112.stdout:[query_scrub_seq]=null 2024-09-13T16:05:53.796 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_pg_state]='active+clean' 2024-09-13T16:05:53.796 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_state_has_scrubbing]=false 2024-09-13T16:05:53.796 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_last_duration]=0 2024-09-13T16:05:53.796 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-13T16:05:53.796 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule_at]='2024-09-14T16:05:44.046351+0000' 2024-09-13T16:05:53.797 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_is_future]=true 2024-09-13T16:05:53.797 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_vs_date]=true 2024-09-13T16:05:53.797 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_reported_epoch]=17 2024-09-13T16:05:53.797 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_seq]=15 2024-09-13T16:05:53.797 INFO:tasks.workunit.client.0.smithi112.stdout:) 2024-09-13T16:05:53.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:579: TEST_dump_scrub_schedule: schedule_against_expected sched_data expct_starting initial 2024-09-13T16:05:53.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:178: schedule_against_expected: local -n dict=sched_data 2024-09-13T16:05:53.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:179: schedule_against_expected: local -n ep=expct_starting 2024-09-13T16:05:53.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:180: schedule_against_expected: local extr_dbg=1 2024-09-13T16:05:53.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:183: schedule_against_expected: local saved_echo_flag=x 2024-09-13T16:05:53.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:184: schedule_against_expected: set +x 2024-09-13T16:05:53.826 INFO:tasks.workunit.client.0.smithi112.stdout:-- - comparing: 2024-09-13T16:05:53.826 INFO:tasks.workunit.client.0.smithi112.stdout:key is query_schedule expected: scrub scheduled in actual: scrub scheduled 2024-09-13T16:05:53.826 INFO:tasks.workunit.client.0.smithi112.stdout:key is query_is_future expected: true in actual: true 2024-09-13T16:05:53.826 INFO:tasks.workunit.client.0.smithi112.stdout:key is query_active expected: false in actual: false 2024-09-13T16:05:53.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:203: schedule_against_expected: return 0 2024-09-13T16:05:53.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:580: TEST_dump_scrub_schedule: (( 0 == 0 )) 2024-09-13T16:05:53.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:581: TEST_dump_scrub_schedule: echo 'last-scrub --- ' 0x0 2024-09-13T16:05:53.827 INFO:tasks.workunit.client.0.smithi112.stdout:last-scrub --- 0x0 2024-09-13T16:05:53.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:587: TEST_dump_scrub_schedule: saved_last_stamp=2024-09-13T16:05:44.046351+0000 2024-09-13T16:05:53.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:588: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 0 2024-09-13T16:05:53.960 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T16:05:53.960 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_scrub_sleep = '' " 2024-09-13T16:05:53.960 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:05:53.973 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T16:05:53.973 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_scrub_sleep = '' " 2024-09-13T16:05:53.974 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:05:53.986 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2: { 2024-09-13T16:05:53.987 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_scrub_sleep = '' " 2024-09-13T16:05:53.987 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:05:54.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:589: TEST_dump_scrub_schedule: ceph tell 1.0 deep-scrub 2024-09-13T16:05:54.170 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:05:54.170 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T16:05:54.170 INFO:tasks.workunit.client.0.smithi112.stdout: "must": true, 2024-09-13T16:05:54.170 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "0.000000" 2024-09-13T16:05:54.170 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:05:54.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:594: TEST_dump_scrub_schedule: sleep 5 2024-09-13T16:05:59.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:595: TEST_dump_scrub_schedule: sched_data=() 2024-09-13T16:05:59.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:596: TEST_dump_scrub_schedule: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2024-09-13T16:05:59.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:596: TEST_dump_scrub_schedule: declare -A expct_qry_duration 2024-09-13T16:05:59.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:597: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-13T16:05:44.046351+0000 expct_qry_duration 'WaitingAfterScrub ' sched_data 2024-09-13T16:05:59.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2024-09-13T16:05:59.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-13T16:05:59.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-13T16:05:44.046351+0000 2024-09-13T16:05:59.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_qry_duration 2024-09-13T16:05:59.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2024-09-13T16:05:59.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-13T16:05:59.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-13T16:05:59.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-13T16:05:59.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-13T16:05:59.188 INFO:tasks.workunit.client.0.smithi112.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.0 dt:2024-09-13T16:05:44.046351+0000 (10 retries) 2024-09-13T16:06:00.006 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:06:00.024 INFO:tasks.workunit.client.0.smithi112.stdout:from pg dump pg: { 2024-09-13T16:06:00.025 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_pg_state": "active+clean", 2024-09-13T16:06:00.025 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_state_has_scrubbing": false, 2024-09-13T16:06:00.025 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_last_duration": 1, 2024-09-13T16:06:00.025 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-13T16:06:00.025 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule_at": "2024-09-14T16:05:54.504667+0000", 2024-09-13T16:06:00.025 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_is_future": true, 2024-09-13T16:06:00.025 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_vs_date": true, 2024-09-13T16:06:00.025 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_reported_epoch": 17, 2024-09-13T16:06:00.025 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_seq": 40 2024-09-13T16:06:00.025 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:00.025 INFO:tasks.workunit.client.0.smithi112.stdout:query output: 2024-09-13T16:06:00.169 INFO:tasks.workunit.client.0.smithi112.stdout: "scrubber": { 2024-09-13T16:06:00.170 INFO:tasks.workunit.client.0.smithi112.stdout: "active": true, 2024-09-13T16:06:00.170 INFO:tasks.workunit.client.0.smithi112.stdout: "epoch_start": "15", 2024-09-13T16:06:00.170 INFO:tasks.workunit.client.0.smithi112.stdout: "start": "MIN", 2024-09-13T16:06:00.170 INFO:tasks.workunit.client.0.smithi112.stdout: "end": "MIN", 2024-09-13T16:06:00.170 INFO:tasks.workunit.client.0.smithi112.stdout: "max_end": "MIN", 2024-09-13T16:06:00.170 INFO:tasks.workunit.client.0.smithi112.stdout: "subset_last_update": "0'0", 2024-09-13T16:06:00.170 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T16:06:00.170 INFO:tasks.workunit.client.0.smithi112.stdout: "req_scrub": false, 2024-09-13T16:06:00.170 INFO:tasks.workunit.client.0.smithi112.stdout: "auto_repair": false, 2024-09-13T16:06:00.170 INFO:tasks.workunit.client.0.smithi112.stdout: "check_repair": false, 2024-09-13T16:06:00.170 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_scrub_on_error": false, 2024-09-13T16:06:00.171 INFO:tasks.workunit.client.0.smithi112.stdout: "priority": 0, 2024-09-13T16:06:00.171 INFO:tasks.workunit.client.0.smithi112.stdout: "shallow_errors": 0, 2024-09-13T16:06:00.171 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_errors": 0, 2024-09-13T16:06:00.171 INFO:tasks.workunit.client.0.smithi112.stdout: "fixed": 0, 2024-09-13T16:06:00.171 INFO:tasks.workunit.client.0.smithi112.stdout: "waiting_on_whom": [], 2024-09-13T16:06:00.171 INFO:tasks.workunit.client.0.smithi112.stdout: "schedule": "scrubbing" 2024-09-13T16:06:00.171 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:00.171 INFO:tasks.workunit.client.0.smithi112.stdout: "agent_state": {} 2024-09-13T16:06:00.339 INFO:tasks.workunit.client.0.smithi112.stdout:( 2024-09-13T16:06:00.339 INFO:tasks.workunit.client.0.smithi112.stdout:[query_epoch]=17 2024-09-13T16:06:00.339 INFO:tasks.workunit.client.0.smithi112.stdout:[query_seq]=41 2024-09-13T16:06:00.340 INFO:tasks.workunit.client.0.smithi112.stdout:[query_active]=true 2024-09-13T16:06:00.340 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule]='scrubbing' 2024-09-13T16:06:00.340 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule_at]='0' 2024-09-13T16:06:00.340 INFO:tasks.workunit.client.0.smithi112.stdout:[query_target_at]='0' 2024-09-13T16:06:00.340 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_duration]=1 2024-09-13T16:06:00.340 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_stamp]='2024-09-13T16:05:54.504667+0000' 2024-09-13T16:06:00.340 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_scrub]='17x15' 2024-09-13T16:06:00.340 INFO:tasks.workunit.client.0.smithi112.stdout:[query_is_future]=false 2024-09-13T16:06:00.340 INFO:tasks.workunit.client.0.smithi112.stdout:[query_vs_date]=false 2024-09-13T16:06:00.340 INFO:tasks.workunit.client.0.smithi112.stdout:[query_scrub_seq]=null 2024-09-13T16:06:00.340 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_pg_state]='active+clean' 2024-09-13T16:06:00.340 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_state_has_scrubbing]=false 2024-09-13T16:06:00.340 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_last_duration]=1 2024-09-13T16:06:00.341 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-13T16:06:00.341 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule_at]='2024-09-14T16:05:54.504667+0000' 2024-09-13T16:06:00.341 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_is_future]=true 2024-09-13T16:06:00.341 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_vs_date]=true 2024-09-13T16:06:00.341 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_reported_epoch]=17 2024-09-13T16:06:00.341 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_seq]=40 2024-09-13T16:06:00.341 INFO:tasks.workunit.client.0.smithi112.stdout:) 2024-09-13T16:06:00.372 INFO:tasks.workunit.client.0.smithi112.stdout:--> loop: 1 ~ true / 41 / 40 / false / 2024-09-13T16:05:54.504667+0000 / scrubbing %%% query_last_duration_neg query_last_duration 2024-09-13T16:06:00.372 INFO:tasks.workunit.client.0.smithi112.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 1 2024-09-13T16:06:00.372 INFO:tasks.workunit.client.0.smithi112.stdout:WaitingAfterScrub - 'query_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-09-13T16:06:00.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2024-09-13T16:06:00.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:599: TEST_dump_scrub_schedule: sched_data=() 2024-09-13T16:06:00.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:600: TEST_dump_scrub_schedule: expct_dmp_duration=(['dmp_last_duration']='0' ['dmp_last_duration_neg']='not0') 2024-09-13T16:06:00.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:600: TEST_dump_scrub_schedule: declare -A expct_dmp_duration 2024-09-13T16:06:00.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:601: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-13T16:05:44.046351+0000 expct_dmp_duration 'WaitingAfterScrub_dmp ' sched_data 2024-09-13T16:06:00.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2024-09-13T16:06:00.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-13T16:06:00.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-13T16:05:44.046351+0000 2024-09-13T16:06:00.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_dmp_duration 2024-09-13T16:06:00.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2024-09-13T16:06:00.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-13T16:06:00.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-13T16:06:00.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-13T16:06:00.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-13T16:06:00.375 INFO:tasks.workunit.client.0.smithi112.stdout:waiting for any condition (WaitingAfterScrub_dmp ): pg:1.0 dt:2024-09-13T16:05:44.046351+0000 (10 retries) 2024-09-13T16:06:01.219 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:06:01.235 INFO:tasks.workunit.client.0.smithi112.stdout:from pg dump pg: { 2024-09-13T16:06:01.235 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_pg_state": "active+clean", 2024-09-13T16:06:01.235 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_state_has_scrubbing": false, 2024-09-13T16:06:01.235 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_last_duration": 1, 2024-09-13T16:06:01.235 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-13T16:06:01.235 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule_at": "2024-09-14T16:05:54.504667+0000", 2024-09-13T16:06:01.235 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_is_future": true, 2024-09-13T16:06:01.236 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_vs_date": true, 2024-09-13T16:06:01.236 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_reported_epoch": 17, 2024-09-13T16:06:01.236 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_seq": 40 2024-09-13T16:06:01.236 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:01.236 INFO:tasks.workunit.client.0.smithi112.stdout:query output: 2024-09-13T16:06:01.415 INFO:tasks.workunit.client.0.smithi112.stdout: "scrubber": { 2024-09-13T16:06:01.415 INFO:tasks.workunit.client.0.smithi112.stdout: "active": true, 2024-09-13T16:06:01.415 INFO:tasks.workunit.client.0.smithi112.stdout: "epoch_start": "15", 2024-09-13T16:06:01.415 INFO:tasks.workunit.client.0.smithi112.stdout: "start": "MIN", 2024-09-13T16:06:01.415 INFO:tasks.workunit.client.0.smithi112.stdout: "end": "MIN", 2024-09-13T16:06:01.415 INFO:tasks.workunit.client.0.smithi112.stdout: "max_end": "MIN", 2024-09-13T16:06:01.415 INFO:tasks.workunit.client.0.smithi112.stdout: "subset_last_update": "0'0", 2024-09-13T16:06:01.415 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T16:06:01.415 INFO:tasks.workunit.client.0.smithi112.stdout: "req_scrub": false, 2024-09-13T16:06:01.415 INFO:tasks.workunit.client.0.smithi112.stdout: "auto_repair": false, 2024-09-13T16:06:01.416 INFO:tasks.workunit.client.0.smithi112.stdout: "check_repair": false, 2024-09-13T16:06:01.416 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_scrub_on_error": false, 2024-09-13T16:06:01.416 INFO:tasks.workunit.client.0.smithi112.stdout: "priority": 0, 2024-09-13T16:06:01.416 INFO:tasks.workunit.client.0.smithi112.stdout: "shallow_errors": 0, 2024-09-13T16:06:01.416 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_errors": 0, 2024-09-13T16:06:01.416 INFO:tasks.workunit.client.0.smithi112.stdout: "fixed": 0, 2024-09-13T16:06:01.416 INFO:tasks.workunit.client.0.smithi112.stdout: "waiting_on_whom": [], 2024-09-13T16:06:01.416 INFO:tasks.workunit.client.0.smithi112.stdout: "schedule": "scrubbing" 2024-09-13T16:06:01.416 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:01.416 INFO:tasks.workunit.client.0.smithi112.stdout: "agent_state": {} 2024-09-13T16:06:01.588 INFO:tasks.workunit.client.0.smithi112.stdout:( 2024-09-13T16:06:01.588 INFO:tasks.workunit.client.0.smithi112.stdout:[query_epoch]=17 2024-09-13T16:06:01.588 INFO:tasks.workunit.client.0.smithi112.stdout:[query_seq]=41 2024-09-13T16:06:01.588 INFO:tasks.workunit.client.0.smithi112.stdout:[query_active]=true 2024-09-13T16:06:01.588 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule]='scrubbing' 2024-09-13T16:06:01.589 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule_at]='0' 2024-09-13T16:06:01.589 INFO:tasks.workunit.client.0.smithi112.stdout:[query_target_at]='0' 2024-09-13T16:06:01.589 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_duration]=1 2024-09-13T16:06:01.589 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_stamp]='2024-09-13T16:05:54.504667+0000' 2024-09-13T16:06:01.589 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_scrub]='17x15' 2024-09-13T16:06:01.589 INFO:tasks.workunit.client.0.smithi112.stdout:[query_is_future]=false 2024-09-13T16:06:01.589 INFO:tasks.workunit.client.0.smithi112.stdout:[query_vs_date]=false 2024-09-13T16:06:01.589 INFO:tasks.workunit.client.0.smithi112.stdout:[query_scrub_seq]=null 2024-09-13T16:06:01.589 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_pg_state]='active+clean' 2024-09-13T16:06:01.589 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_state_has_scrubbing]=false 2024-09-13T16:06:01.589 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_last_duration]=1 2024-09-13T16:06:01.589 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-13T16:06:01.589 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule_at]='2024-09-14T16:05:54.504667+0000' 2024-09-13T16:06:01.589 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_is_future]=true 2024-09-13T16:06:01.590 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_vs_date]=true 2024-09-13T16:06:01.590 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_reported_epoch]=17 2024-09-13T16:06:01.590 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_seq]=40 2024-09-13T16:06:01.590 INFO:tasks.workunit.client.0.smithi112.stdout:) 2024-09-13T16:06:01.624 INFO:tasks.workunit.client.0.smithi112.stdout:--> loop: 1 ~ true / 41 / 40 / false / 2024-09-13T16:05:54.504667+0000 / scrubbing %%% dmp_last_duration dmp_last_duration_neg 2024-09-13T16:06:01.624 INFO:tasks.workunit.client.0.smithi112.stdout:key is dmp_last_duration: negation:1 # expected: 0 # in actual: 1 2024-09-13T16:06:01.624 INFO:tasks.workunit.client.0.smithi112.stdout:WaitingAfterScrub_dmp - 'dmp_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-09-13T16:06:01.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2024-09-13T16:06:01.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:603: TEST_dump_scrub_schedule: sleep 2 2024-09-13T16:06:03.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:610: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2024-09-13T16:06:03.766 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T16:06:03.766 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) " 2024-09-13T16:06:03.766 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:03.779 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T16:06:03.779 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) " 2024-09-13T16:06:03.779 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:03.793 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2: { 2024-09-13T16:06:03.793 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) " 2024-09-13T16:06:03.793 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:03.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:611: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 2.0 2024-09-13T16:06:03.958 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0: { 2024-09-13T16:06:03.959 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_scrub_sleep = '' " 2024-09-13T16:06:03.959 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:03.972 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1: { 2024-09-13T16:06:03.972 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_scrub_sleep = '' " 2024-09-13T16:06:03.972 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:03.984 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2: { 2024-09-13T16:06:03.985 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_scrub_sleep = '' " 2024-09-13T16:06:03.985 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:03.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:612: TEST_dump_scrub_schedule: ceph osd set noscrub 2024-09-13T16:06:04.448 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T16:06:04.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:613: TEST_dump_scrub_schedule: sleep 2 2024-09-13T16:06:06.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:614: TEST_dump_scrub_schedule: saved_last_stamp=2024-09-13T16:05:54.504667+0000 2024-09-13T16:06:06.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:616: TEST_dump_scrub_schedule: ceph tell 1.0 schedule-scrub 2024-09-13T16:06:06.607 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:06:06.607 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T16:06:06.607 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T16:06:06.607 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-06T16:04:26.608353+0000" 2024-09-13T16:06:06.607 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:06.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:617: TEST_dump_scrub_schedule: sleep 1 2024-09-13T16:06:07.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:618: TEST_dump_scrub_schedule: sched_data=() 2024-09-13T16:06:07.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:619: TEST_dump_scrub_schedule: expct_scrub_peri_sched=(['query_is_future']='false') 2024-09-13T16:06:07.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:619: TEST_dump_scrub_schedule: declare -A expct_scrub_peri_sched 2024-09-13T16:06:07.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:620: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-13T16:05:54.504667+0000 expct_scrub_peri_sched waitingBeingScheduled sched_data 2024-09-13T16:06:07.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2024-09-13T16:06:07.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-13T16:06:07.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-13T16:05:54.504667+0000 2024-09-13T16:06:07.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_scrub_peri_sched 2024-09-13T16:06:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2024-09-13T16:06:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-13T16:06:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-13T16:06:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-13T16:06:07.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-13T16:06:07.626 INFO:tasks.workunit.client.0.smithi112.stdout:waiting for any condition (waitingBeingScheduled): pg:1.0 dt:2024-09-13T16:05:54.504667+0000 (10 retries) 2024-09-13T16:06:08.434 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:06:08.450 INFO:tasks.workunit.client.0.smithi112.stdout:from pg dump pg: { 2024-09-13T16:06:08.450 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_pg_state": "active+clean", 2024-09-13T16:06:08.451 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_state_has_scrubbing": false, 2024-09-13T16:06:08.451 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_last_duration": 1, 2024-09-13T16:06:08.451 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-13T16:06:08.451 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule_at": "2024-09-14T16:05:54.504667+0000", 2024-09-13T16:06:08.451 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_is_future": true, 2024-09-13T16:06:08.451 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_vs_date": true, 2024-09-13T16:06:08.451 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_reported_epoch": 17, 2024-09-13T16:06:08.451 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_seq": 41 2024-09-13T16:06:08.451 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:08.451 INFO:tasks.workunit.client.0.smithi112.stdout:query output: 2024-09-13T16:06:08.596 INFO:tasks.workunit.client.0.smithi112.stdout: "scrubber": { 2024-09-13T16:06:08.597 INFO:tasks.workunit.client.0.smithi112.stdout: "active": true, 2024-09-13T16:06:08.597 INFO:tasks.workunit.client.0.smithi112.stdout: "epoch_start": "15", 2024-09-13T16:06:08.597 INFO:tasks.workunit.client.0.smithi112.stdout: "start": "MIN", 2024-09-13T16:06:08.597 INFO:tasks.workunit.client.0.smithi112.stdout: "end": "MIN", 2024-09-13T16:06:08.597 INFO:tasks.workunit.client.0.smithi112.stdout: "max_end": "MIN", 2024-09-13T16:06:08.597 INFO:tasks.workunit.client.0.smithi112.stdout: "subset_last_update": "0'0", 2024-09-13T16:06:08.597 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T16:06:08.597 INFO:tasks.workunit.client.0.smithi112.stdout: "req_scrub": false, 2024-09-13T16:06:08.597 INFO:tasks.workunit.client.0.smithi112.stdout: "auto_repair": false, 2024-09-13T16:06:08.597 INFO:tasks.workunit.client.0.smithi112.stdout: "check_repair": false, 2024-09-13T16:06:08.597 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_scrub_on_error": false, 2024-09-13T16:06:08.597 INFO:tasks.workunit.client.0.smithi112.stdout: "priority": 0, 2024-09-13T16:06:08.597 INFO:tasks.workunit.client.0.smithi112.stdout: "shallow_errors": 0, 2024-09-13T16:06:08.598 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_errors": 0, 2024-09-13T16:06:08.598 INFO:tasks.workunit.client.0.smithi112.stdout: "fixed": 0, 2024-09-13T16:06:08.598 INFO:tasks.workunit.client.0.smithi112.stdout: "waiting_on_whom": [], 2024-09-13T16:06:08.598 INFO:tasks.workunit.client.0.smithi112.stdout: "schedule": "scrubbing" 2024-09-13T16:06:08.598 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:08.598 INFO:tasks.workunit.client.0.smithi112.stdout: "agent_state": {} 2024-09-13T16:06:08.779 INFO:tasks.workunit.client.0.smithi112.stdout:( 2024-09-13T16:06:08.779 INFO:tasks.workunit.client.0.smithi112.stdout:[query_epoch]=18 2024-09-13T16:06:08.779 INFO:tasks.workunit.client.0.smithi112.stdout:[query_seq]=47 2024-09-13T16:06:08.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_active]=true 2024-09-13T16:06:08.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule]='scrubbing' 2024-09-13T16:06:08.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule_at]='0' 2024-09-13T16:06:08.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_target_at]='0' 2024-09-13T16:06:08.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_duration]=1 2024-09-13T16:06:08.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_stamp]='2024-09-06T16:04:26.608353+0000' 2024-09-13T16:06:08.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_scrub]='17x15' 2024-09-13T16:06:08.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_is_future]=false 2024-09-13T16:06:08.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_vs_date]=false 2024-09-13T16:06:08.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_scrub_seq]=null 2024-09-13T16:06:08.780 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_pg_state]='active+clean' 2024-09-13T16:06:08.780 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_state_has_scrubbing]=false 2024-09-13T16:06:08.780 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_last_duration]=1 2024-09-13T16:06:08.781 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-13T16:06:08.781 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule_at]='2024-09-14T16:05:54.504667+0000' 2024-09-13T16:06:08.781 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_is_future]=true 2024-09-13T16:06:08.781 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_vs_date]=true 2024-09-13T16:06:08.781 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_reported_epoch]=17 2024-09-13T16:06:08.781 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_seq]=41 2024-09-13T16:06:08.781 INFO:tasks.workunit.client.0.smithi112.stdout:) 2024-09-13T16:06:08.816 INFO:tasks.workunit.client.0.smithi112.stdout:--> loop: 1 ~ true / 47 / 41 / false / 2024-09-06T16:04:26.608353+0000 / scrubbing %%% query_is_future 2024-09-13T16:06:08.816 INFO:tasks.workunit.client.0.smithi112.stdout:key is query_is_future: negation:0 # expected: false # in actual: false 2024-09-13T16:06:08.816 INFO:tasks.workunit.client.0.smithi112.stdout:waitingBeingScheduled - 'query_is_future' actual value (false) matches expected (false) (negation: 0) 2024-09-13T16:06:08.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2024-09-13T16:06:08.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:632: TEST_dump_scrub_schedule: saved_last_stamp=2024-09-06T16:04:26.608353+0000 2024-09-13T16:06:08.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:633: TEST_dump_scrub_schedule: ceph osd unset noscrub 2024-09-13T16:06:09.394 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is unset 2024-09-13T16:06:09.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:635: TEST_dump_scrub_schedule: cond_active=(['query_active']='true') 2024-09-13T16:06:09.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:635: TEST_dump_scrub_schedule: declare -A cond_active 2024-09-13T16:06:09.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:636: TEST_dump_scrub_schedule: sched_data=() 2024-09-13T16:06:09.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:637: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-06T16:04:26.608353+0000 cond_active 'WaitingActive ' sched_data 2024-09-13T16:06:09.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2024-09-13T16:06:09.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-13T16:06:09.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-06T16:04:26.608353+0000 2024-09-13T16:06:09.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=cond_active 2024-09-13T16:06:09.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2024-09-13T16:06:09.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-13T16:06:09.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-13T16:06:09.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-13T16:06:09.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-13T16:06:09.419 INFO:tasks.workunit.client.0.smithi112.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2024-09-06T16:04:26.608353+0000 (10 retries) 2024-09-13T16:06:10.225 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:06:10.241 INFO:tasks.workunit.client.0.smithi112.stdout:from pg dump pg: { 2024-09-13T16:06:10.241 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_pg_state": "active+clean", 2024-09-13T16:06:10.242 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_state_has_scrubbing": false, 2024-09-13T16:06:10.242 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_last_duration": 1, 2024-09-13T16:06:10.242 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-13T16:06:10.242 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule_at": "2024-09-13T16:06:12.258306+0000", 2024-09-13T16:06:10.242 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_is_future": true, 2024-09-13T16:06:10.242 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_vs_date": true, 2024-09-13T16:06:10.242 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_reported_epoch": 18, 2024-09-13T16:06:10.242 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_seq": 46 2024-09-13T16:06:10.242 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:10.242 INFO:tasks.workunit.client.0.smithi112.stdout:query output: 2024-09-13T16:06:10.425 INFO:tasks.workunit.client.0.smithi112.stdout: "scrubber": { 2024-09-13T16:06:10.426 INFO:tasks.workunit.client.0.smithi112.stdout: "active": true, 2024-09-13T16:06:10.426 INFO:tasks.workunit.client.0.smithi112.stdout: "epoch_start": "15", 2024-09-13T16:06:10.426 INFO:tasks.workunit.client.0.smithi112.stdout: "start": "MIN", 2024-09-13T16:06:10.426 INFO:tasks.workunit.client.0.smithi112.stdout: "end": "MIN", 2024-09-13T16:06:10.426 INFO:tasks.workunit.client.0.smithi112.stdout: "max_end": "MIN", 2024-09-13T16:06:10.426 INFO:tasks.workunit.client.0.smithi112.stdout: "subset_last_update": "0'0", 2024-09-13T16:06:10.426 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T16:06:10.426 INFO:tasks.workunit.client.0.smithi112.stdout: "req_scrub": false, 2024-09-13T16:06:10.426 INFO:tasks.workunit.client.0.smithi112.stdout: "auto_repair": false, 2024-09-13T16:06:10.426 INFO:tasks.workunit.client.0.smithi112.stdout: "check_repair": false, 2024-09-13T16:06:10.427 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_scrub_on_error": false, 2024-09-13T16:06:10.427 INFO:tasks.workunit.client.0.smithi112.stdout: "priority": 0, 2024-09-13T16:06:10.427 INFO:tasks.workunit.client.0.smithi112.stdout: "shallow_errors": 0, 2024-09-13T16:06:10.427 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_errors": 0, 2024-09-13T16:06:10.427 INFO:tasks.workunit.client.0.smithi112.stdout: "fixed": 0, 2024-09-13T16:06:10.427 INFO:tasks.workunit.client.0.smithi112.stdout: "waiting_on_whom": [], 2024-09-13T16:06:10.427 INFO:tasks.workunit.client.0.smithi112.stdout: "schedule": "scrubbing" 2024-09-13T16:06:10.427 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:10.427 INFO:tasks.workunit.client.0.smithi112.stdout: "agent_state": {} 2024-09-13T16:06:10.597 INFO:tasks.workunit.client.0.smithi112.stdout:( 2024-09-13T16:06:10.597 INFO:tasks.workunit.client.0.smithi112.stdout:[query_epoch]=19 2024-09-13T16:06:10.597 INFO:tasks.workunit.client.0.smithi112.stdout:[query_seq]=49 2024-09-13T16:06:10.597 INFO:tasks.workunit.client.0.smithi112.stdout:[query_active]=true 2024-09-13T16:06:10.597 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule]='scrubbing' 2024-09-13T16:06:10.597 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule_at]='0' 2024-09-13T16:06:10.597 INFO:tasks.workunit.client.0.smithi112.stdout:[query_target_at]='0' 2024-09-13T16:06:10.597 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_duration]=1 2024-09-13T16:06:10.597 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_stamp]='2024-09-06T16:04:26.608353+0000' 2024-09-13T16:06:10.597 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_scrub]='17x15' 2024-09-13T16:06:10.598 INFO:tasks.workunit.client.0.smithi112.stdout:[query_is_future]=false 2024-09-13T16:06:10.598 INFO:tasks.workunit.client.0.smithi112.stdout:[query_vs_date]=false 2024-09-13T16:06:10.598 INFO:tasks.workunit.client.0.smithi112.stdout:[query_scrub_seq]=null 2024-09-13T16:06:10.598 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_pg_state]='active+clean' 2024-09-13T16:06:10.598 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_state_has_scrubbing]=false 2024-09-13T16:06:10.598 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_last_duration]=1 2024-09-13T16:06:10.598 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-13T16:06:10.598 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule_at]='2024-09-13T16:06:12.258306+0000' 2024-09-13T16:06:10.598 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_is_future]=true 2024-09-13T16:06:10.598 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_vs_date]=true 2024-09-13T16:06:10.598 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_reported_epoch]=18 2024-09-13T16:06:10.598 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_seq]=46 2024-09-13T16:06:10.598 INFO:tasks.workunit.client.0.smithi112.stdout:) 2024-09-13T16:06:10.629 INFO:tasks.workunit.client.0.smithi112.stdout:--> loop: 1 ~ true / 49 / 46 / false / 2024-09-06T16:04:26.608353+0000 / scrubbing %%% query_active 2024-09-13T16:06:10.629 INFO:tasks.workunit.client.0.smithi112.stdout:key is query_active: negation:0 # expected: true # in actual: true 2024-09-13T16:06:10.629 INFO:tasks.workunit.client.0.smithi112.stdout:WaitingActive - 'query_active' actual value (true) matches expected (true) (negation: 0) 2024-09-13T16:06:10.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2024-09-13T16:06:10.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_dump_scrub_schedule: cond_active_dmp=(['dmp_state_has_scrubbing']='true' ['query_active']='false') 2024-09-13T16:06:10.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_dump_scrub_schedule: declare -A cond_active_dmp 2024-09-13T16:06:10.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:642: TEST_dump_scrub_schedule: sched_data=() 2024-09-13T16:06:10.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:643: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-09-06T16:04:26.608353+0000 cond_active_dmp 'WaitingActive ' sched_data 2024-09-13T16:06:10.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2024-09-13T16:06:10.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-13T16:06:10.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-06T16:04:26.608353+0000 2024-09-13T16:06:10.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=cond_active_dmp 2024-09-13T16:06:10.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2024-09-13T16:06:10.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-13T16:06:10.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-13T16:06:10.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-13T16:06:10.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-13T16:06:10.632 INFO:tasks.workunit.client.0.smithi112.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2024-09-06T16:04:26.608353+0000 (10 retries) 2024-09-13T16:06:11.446 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:06:11.462 INFO:tasks.workunit.client.0.smithi112.stdout:from pg dump pg: { 2024-09-13T16:06:11.463 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_pg_state": "active+clean", 2024-09-13T16:06:11.463 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_state_has_scrubbing": false, 2024-09-13T16:06:11.463 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_last_duration": 1, 2024-09-13T16:06:11.463 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-13T16:06:11.463 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule_at": "2024-09-13T16:06:12.258306+0000", 2024-09-13T16:06:11.463 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_is_future": true, 2024-09-13T16:06:11.463 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_vs_date": true, 2024-09-13T16:06:11.463 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_reported_epoch": 18, 2024-09-13T16:06:11.463 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_seq": 46 2024-09-13T16:06:11.463 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:11.463 INFO:tasks.workunit.client.0.smithi112.stdout:query output: 2024-09-13T16:06:11.607 INFO:tasks.workunit.client.0.smithi112.stdout: "scrubber": { 2024-09-13T16:06:11.608 INFO:tasks.workunit.client.0.smithi112.stdout: "active": true, 2024-09-13T16:06:11.608 INFO:tasks.workunit.client.0.smithi112.stdout: "epoch_start": "15", 2024-09-13T16:06:11.608 INFO:tasks.workunit.client.0.smithi112.stdout: "start": "MIN", 2024-09-13T16:06:11.608 INFO:tasks.workunit.client.0.smithi112.stdout: "end": "MIN", 2024-09-13T16:06:11.608 INFO:tasks.workunit.client.0.smithi112.stdout: "max_end": "MIN", 2024-09-13T16:06:11.608 INFO:tasks.workunit.client.0.smithi112.stdout: "subset_last_update": "0'0", 2024-09-13T16:06:11.608 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T16:06:11.608 INFO:tasks.workunit.client.0.smithi112.stdout: "req_scrub": false, 2024-09-13T16:06:11.608 INFO:tasks.workunit.client.0.smithi112.stdout: "auto_repair": false, 2024-09-13T16:06:11.608 INFO:tasks.workunit.client.0.smithi112.stdout: "check_repair": false, 2024-09-13T16:06:11.608 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_scrub_on_error": false, 2024-09-13T16:06:11.608 INFO:tasks.workunit.client.0.smithi112.stdout: "priority": 0, 2024-09-13T16:06:11.608 INFO:tasks.workunit.client.0.smithi112.stdout: "shallow_errors": 0, 2024-09-13T16:06:11.609 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_errors": 0, 2024-09-13T16:06:11.609 INFO:tasks.workunit.client.0.smithi112.stdout: "fixed": 0, 2024-09-13T16:06:11.609 INFO:tasks.workunit.client.0.smithi112.stdout: "waiting_on_whom": [], 2024-09-13T16:06:11.609 INFO:tasks.workunit.client.0.smithi112.stdout: "schedule": "scrubbing" 2024-09-13T16:06:11.609 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:11.609 INFO:tasks.workunit.client.0.smithi112.stdout: "agent_state": {} 2024-09-13T16:06:11.779 INFO:tasks.workunit.client.0.smithi112.stdout:( 2024-09-13T16:06:11.779 INFO:tasks.workunit.client.0.smithi112.stdout:[query_epoch]=19 2024-09-13T16:06:11.779 INFO:tasks.workunit.client.0.smithi112.stdout:[query_seq]=49 2024-09-13T16:06:11.779 INFO:tasks.workunit.client.0.smithi112.stdout:[query_active]=true 2024-09-13T16:06:11.779 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule]='scrubbing' 2024-09-13T16:06:11.779 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule_at]='0' 2024-09-13T16:06:11.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_target_at]='0' 2024-09-13T16:06:11.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_duration]=1 2024-09-13T16:06:11.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_stamp]='2024-09-06T16:04:26.608353+0000' 2024-09-13T16:06:11.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_scrub]='17x15' 2024-09-13T16:06:11.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_is_future]=false 2024-09-13T16:06:11.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_vs_date]=false 2024-09-13T16:06:11.780 INFO:tasks.workunit.client.0.smithi112.stdout:[query_scrub_seq]=null 2024-09-13T16:06:11.780 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_pg_state]='active+clean' 2024-09-13T16:06:11.780 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_state_has_scrubbing]=false 2024-09-13T16:06:11.780 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_last_duration]=1 2024-09-13T16:06:11.780 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-13T16:06:11.780 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule_at]='2024-09-13T16:06:12.258306+0000' 2024-09-13T16:06:11.780 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_is_future]=true 2024-09-13T16:06:11.781 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_vs_date]=true 2024-09-13T16:06:11.781 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_reported_epoch]=18 2024-09-13T16:06:11.781 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_seq]=46 2024-09-13T16:06:11.781 INFO:tasks.workunit.client.0.smithi112.stdout:) 2024-09-13T16:06:11.811 INFO:tasks.workunit.client.0.smithi112.stdout:--> loop: 1 ~ true / 49 / 46 / false / 2024-09-06T16:04:26.608353+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-13T16:06:11.811 INFO:tasks.workunit.client.0.smithi112.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-13T16:06:11.812 INFO:tasks.workunit.client.0.smithi112.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-13T16:06:12.678 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:06:12.695 INFO:tasks.workunit.client.0.smithi112.stdout:from pg dump pg: { 2024-09-13T16:06:12.695 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_pg_state": "active+clean", 2024-09-13T16:06:12.695 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_state_has_scrubbing": false, 2024-09-13T16:06:12.695 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_last_duration": 1, 2024-09-13T16:06:12.695 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-13T16:06:12.695 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule_at": "2024-09-13T16:06:12.258306+0000", 2024-09-13T16:06:12.695 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_is_future": true, 2024-09-13T16:06:12.696 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_vs_date": true, 2024-09-13T16:06:12.696 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_reported_epoch": 18, 2024-09-13T16:06:12.696 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_seq": 46 2024-09-13T16:06:12.696 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:12.696 INFO:tasks.workunit.client.0.smithi112.stdout:query output: 2024-09-13T16:06:12.837 INFO:tasks.workunit.client.0.smithi112.stdout: "scrubber": { 2024-09-13T16:06:12.837 INFO:tasks.workunit.client.0.smithi112.stdout: "active": true, 2024-09-13T16:06:12.837 INFO:tasks.workunit.client.0.smithi112.stdout: "epoch_start": "15", 2024-09-13T16:06:12.837 INFO:tasks.workunit.client.0.smithi112.stdout: "start": "1:00000000::::0", 2024-09-13T16:06:12.837 INFO:tasks.workunit.client.0.smithi112.stdout: "end": "MIN", 2024-09-13T16:06:12.837 INFO:tasks.workunit.client.0.smithi112.stdout: "max_end": "MIN", 2024-09-13T16:06:12.837 INFO:tasks.workunit.client.0.smithi112.stdout: "subset_last_update": "0'0", 2024-09-13T16:06:12.837 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T16:06:12.837 INFO:tasks.workunit.client.0.smithi112.stdout: "req_scrub": false, 2024-09-13T16:06:12.837 INFO:tasks.workunit.client.0.smithi112.stdout: "auto_repair": false, 2024-09-13T16:06:12.837 INFO:tasks.workunit.client.0.smithi112.stdout: "check_repair": false, 2024-09-13T16:06:12.837 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_scrub_on_error": false, 2024-09-13T16:06:12.838 INFO:tasks.workunit.client.0.smithi112.stdout: "priority": 5, 2024-09-13T16:06:12.838 INFO:tasks.workunit.client.0.smithi112.stdout: "shallow_errors": 0, 2024-09-13T16:06:12.838 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_errors": 0, 2024-09-13T16:06:12.838 INFO:tasks.workunit.client.0.smithi112.stdout: "fixed": 0, 2024-09-13T16:06:12.838 INFO:tasks.workunit.client.0.smithi112.stdout: "waiting_on_whom": [], 2024-09-13T16:06:12.838 INFO:tasks.workunit.client.0.smithi112.stdout: "schedule": "scrubbing" 2024-09-13T16:06:12.838 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:12.838 INFO:tasks.workunit.client.0.smithi112.stdout: "agent_state": {} 2024-09-13T16:06:12.995 INFO:tasks.workunit.client.0.smithi112.stdout:( 2024-09-13T16:06:12.995 INFO:tasks.workunit.client.0.smithi112.stdout:[query_epoch]=19 2024-09-13T16:06:12.995 INFO:tasks.workunit.client.0.smithi112.stdout:[query_seq]=52 2024-09-13T16:06:12.995 INFO:tasks.workunit.client.0.smithi112.stdout:[query_active]=true 2024-09-13T16:06:12.995 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule]='scrubbing' 2024-09-13T16:06:12.995 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule_at]='0' 2024-09-13T16:06:12.995 INFO:tasks.workunit.client.0.smithi112.stdout:[query_target_at]='0' 2024-09-13T16:06:12.995 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_duration]=1 2024-09-13T16:06:12.996 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_stamp]='2024-09-06T16:04:26.608353+0000' 2024-09-13T16:06:12.996 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_scrub]='17x15' 2024-09-13T16:06:12.996 INFO:tasks.workunit.client.0.smithi112.stdout:[query_is_future]=false 2024-09-13T16:06:12.996 INFO:tasks.workunit.client.0.smithi112.stdout:[query_vs_date]=false 2024-09-13T16:06:12.996 INFO:tasks.workunit.client.0.smithi112.stdout:[query_scrub_seq]=null 2024-09-13T16:06:12.996 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_pg_state]='active+clean' 2024-09-13T16:06:12.996 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_state_has_scrubbing]=false 2024-09-13T16:06:12.996 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_last_duration]=1 2024-09-13T16:06:12.996 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-13T16:06:12.996 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule_at]='2024-09-13T16:06:12.258306+0000' 2024-09-13T16:06:12.996 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_is_future]=true 2024-09-13T16:06:12.996 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_vs_date]=true 2024-09-13T16:06:12.996 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_reported_epoch]=18 2024-09-13T16:06:12.996 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_seq]=46 2024-09-13T16:06:12.997 INFO:tasks.workunit.client.0.smithi112.stdout:) 2024-09-13T16:06:13.030 INFO:tasks.workunit.client.0.smithi112.stdout:--> loop: 2 ~ true / 52 / 46 / false / 2024-09-06T16:04:26.608353+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-13T16:06:13.030 INFO:tasks.workunit.client.0.smithi112.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-13T16:06:13.030 INFO:tasks.workunit.client.0.smithi112.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-13T16:06:13.839 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:06:13.857 INFO:tasks.workunit.client.0.smithi112.stdout:from pg dump pg: { 2024-09-13T16:06:13.857 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_pg_state": "active+clean", 2024-09-13T16:06:13.857 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_state_has_scrubbing": false, 2024-09-13T16:06:13.857 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_last_duration": 1, 2024-09-13T16:06:13.857 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-13T16:06:13.857 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule_at": "2024-09-13T16:06:12.258306+0000", 2024-09-13T16:06:13.857 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_is_future": true, 2024-09-13T16:06:13.857 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_vs_date": true, 2024-09-13T16:06:13.857 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_reported_epoch": 18, 2024-09-13T16:06:13.857 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_seq": 46 2024-09-13T16:06:13.857 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:13.858 INFO:tasks.workunit.client.0.smithi112.stdout:query output: 2024-09-13T16:06:13.996 INFO:tasks.workunit.client.0.smithi112.stdout: "scrubber": { 2024-09-13T16:06:13.997 INFO:tasks.workunit.client.0.smithi112.stdout: "active": true, 2024-09-13T16:06:13.997 INFO:tasks.workunit.client.0.smithi112.stdout: "epoch_start": "15", 2024-09-13T16:06:13.997 INFO:tasks.workunit.client.0.smithi112.stdout: "start": "1:00000000::::0", 2024-09-13T16:06:13.997 INFO:tasks.workunit.client.0.smithi112.stdout: "end": "MIN", 2024-09-13T16:06:13.997 INFO:tasks.workunit.client.0.smithi112.stdout: "max_end": "MIN", 2024-09-13T16:06:13.997 INFO:tasks.workunit.client.0.smithi112.stdout: "subset_last_update": "0'0", 2024-09-13T16:06:13.997 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T16:06:13.997 INFO:tasks.workunit.client.0.smithi112.stdout: "req_scrub": false, 2024-09-13T16:06:13.997 INFO:tasks.workunit.client.0.smithi112.stdout: "auto_repair": false, 2024-09-13T16:06:13.997 INFO:tasks.workunit.client.0.smithi112.stdout: "check_repair": false, 2024-09-13T16:06:13.997 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_scrub_on_error": false, 2024-09-13T16:06:13.997 INFO:tasks.workunit.client.0.smithi112.stdout: "priority": 5, 2024-09-13T16:06:13.998 INFO:tasks.workunit.client.0.smithi112.stdout: "shallow_errors": 0, 2024-09-13T16:06:13.998 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_errors": 0, 2024-09-13T16:06:13.998 INFO:tasks.workunit.client.0.smithi112.stdout: "fixed": 0, 2024-09-13T16:06:13.998 INFO:tasks.workunit.client.0.smithi112.stdout: "waiting_on_whom": [], 2024-09-13T16:06:13.998 INFO:tasks.workunit.client.0.smithi112.stdout: "schedule": "scrubbing" 2024-09-13T16:06:13.998 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:13.998 INFO:tasks.workunit.client.0.smithi112.stdout: "agent_state": {} 2024-09-13T16:06:14.158 INFO:tasks.workunit.client.0.smithi112.stdout:( 2024-09-13T16:06:14.158 INFO:tasks.workunit.client.0.smithi112.stdout:[query_epoch]=19 2024-09-13T16:06:14.158 INFO:tasks.workunit.client.0.smithi112.stdout:[query_seq]=53 2024-09-13T16:06:14.158 INFO:tasks.workunit.client.0.smithi112.stdout:[query_active]=true 2024-09-13T16:06:14.158 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule]='scrubbing' 2024-09-13T16:06:14.158 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule_at]='0' 2024-09-13T16:06:14.158 INFO:tasks.workunit.client.0.smithi112.stdout:[query_target_at]='0' 2024-09-13T16:06:14.159 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_duration]=1 2024-09-13T16:06:14.159 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_stamp]='2024-09-06T16:04:26.608353+0000' 2024-09-13T16:06:14.159 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_scrub]='17x15' 2024-09-13T16:06:14.159 INFO:tasks.workunit.client.0.smithi112.stdout:[query_is_future]=false 2024-09-13T16:06:14.159 INFO:tasks.workunit.client.0.smithi112.stdout:[query_vs_date]=false 2024-09-13T16:06:14.159 INFO:tasks.workunit.client.0.smithi112.stdout:[query_scrub_seq]=null 2024-09-13T16:06:14.159 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_pg_state]='active+clean' 2024-09-13T16:06:14.159 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_state_has_scrubbing]=false 2024-09-13T16:06:14.159 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_last_duration]=1 2024-09-13T16:06:14.159 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-13T16:06:14.159 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule_at]='2024-09-13T16:06:12.258306+0000' 2024-09-13T16:06:14.159 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_is_future]=true 2024-09-13T16:06:14.159 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_vs_date]=true 2024-09-13T16:06:14.159 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_reported_epoch]=18 2024-09-13T16:06:14.160 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_seq]=46 2024-09-13T16:06:14.160 INFO:tasks.workunit.client.0.smithi112.stdout:) 2024-09-13T16:06:14.187 INFO:tasks.workunit.client.0.smithi112.stdout:--> loop: 3 ~ true / 53 / 46 / false / 2024-09-06T16:04:26.608353+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-13T16:06:14.187 INFO:tasks.workunit.client.0.smithi112.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-09-13T16:06:14.188 INFO:tasks.workunit.client.0.smithi112.stdout:key is query_active: negation:0 # expected: false # in actual: true 2024-09-13T16:06:15.000 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:06:15.017 INFO:tasks.workunit.client.0.smithi112.stdout:from pg dump pg: { 2024-09-13T16:06:15.017 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_pg_state": "active+clean+scrubbing", 2024-09-13T16:06:15.017 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_state_has_scrubbing": true, 2024-09-13T16:06:15.017 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_last_duration": 1, 2024-09-13T16:06:15.017 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule": "scrubbing for 0s", 2024-09-13T16:06:15.017 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule_at": "0", 2024-09-13T16:06:15.018 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_is_future": false, 2024-09-13T16:06:15.018 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_vs_date": false, 2024-09-13T16:06:15.018 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_reported_epoch": 19, 2024-09-13T16:06:15.018 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_seq": 52 2024-09-13T16:06:15.018 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:15.018 INFO:tasks.workunit.client.0.smithi112.stdout:query output: 2024-09-13T16:06:15.166 INFO:tasks.workunit.client.0.smithi112.stdout: "scrubber": { 2024-09-13T16:06:15.166 INFO:tasks.workunit.client.0.smithi112.stdout: "active": true, 2024-09-13T16:06:15.167 INFO:tasks.workunit.client.0.smithi112.stdout: "epoch_start": "15", 2024-09-13T16:06:15.167 INFO:tasks.workunit.client.0.smithi112.stdout: "start": "MIN", 2024-09-13T16:06:15.167 INFO:tasks.workunit.client.0.smithi112.stdout: "end": "MIN", 2024-09-13T16:06:15.167 INFO:tasks.workunit.client.0.smithi112.stdout: "max_end": "MIN", 2024-09-13T16:06:15.167 INFO:tasks.workunit.client.0.smithi112.stdout: "subset_last_update": "0'0", 2024-09-13T16:06:15.167 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T16:06:15.167 INFO:tasks.workunit.client.0.smithi112.stdout: "req_scrub": false, 2024-09-13T16:06:15.167 INFO:tasks.workunit.client.0.smithi112.stdout: "auto_repair": false, 2024-09-13T16:06:15.167 INFO:tasks.workunit.client.0.smithi112.stdout: "check_repair": false, 2024-09-13T16:06:15.167 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_scrub_on_error": false, 2024-09-13T16:06:15.167 INFO:tasks.workunit.client.0.smithi112.stdout: "priority": 0, 2024-09-13T16:06:15.167 INFO:tasks.workunit.client.0.smithi112.stdout: "shallow_errors": 0, 2024-09-13T16:06:15.167 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_errors": 0, 2024-09-13T16:06:15.167 INFO:tasks.workunit.client.0.smithi112.stdout: "fixed": 0, 2024-09-13T16:06:15.168 INFO:tasks.workunit.client.0.smithi112.stdout: "waiting_on_whom": [], 2024-09-13T16:06:15.168 INFO:tasks.workunit.client.0.smithi112.stdout: "schedule": "scrubbing" 2024-09-13T16:06:15.168 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.168 INFO:tasks.workunit.client.0.smithi112.stdout: "agent_state": {} 2024-09-13T16:06:15.339 INFO:tasks.workunit.client.0.smithi112.stdout:( 2024-09-13T16:06:15.339 INFO:tasks.workunit.client.0.smithi112.stdout:[query_epoch]=19 2024-09-13T16:06:15.339 INFO:tasks.workunit.client.0.smithi112.stdout:[query_seq]=58 2024-09-13T16:06:15.339 INFO:tasks.workunit.client.0.smithi112.stdout:[query_active]=true 2024-09-13T16:06:15.339 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule]='scrubbing' 2024-09-13T16:06:15.339 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule_at]='0' 2024-09-13T16:06:15.339 INFO:tasks.workunit.client.0.smithi112.stdout:[query_target_at]='0' 2024-09-13T16:06:15.339 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_duration]=3 2024-09-13T16:06:15.339 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_stamp]='2024-09-13T16:06:14.359215+0000' 2024-09-13T16:06:15.340 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_scrub]='17x15' 2024-09-13T16:06:15.340 INFO:tasks.workunit.client.0.smithi112.stdout:[query_is_future]=false 2024-09-13T16:06:15.340 INFO:tasks.workunit.client.0.smithi112.stdout:[query_vs_date]=false 2024-09-13T16:06:15.340 INFO:tasks.workunit.client.0.smithi112.stdout:[query_scrub_seq]=null 2024-09-13T16:06:15.340 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_pg_state]='active+clean+scrubbing' 2024-09-13T16:06:15.340 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_state_has_scrubbing]=true 2024-09-13T16:06:15.340 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_last_duration]=1 2024-09-13T16:06:15.340 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule]='scrubbing for 0s' 2024-09-13T16:06:15.340 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule_at]='0' 2024-09-13T16:06:15.340 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_is_future]=false 2024-09-13T16:06:15.340 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_vs_date]=false 2024-09-13T16:06:15.340 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_reported_epoch]=19 2024-09-13T16:06:15.340 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_seq]=52 2024-09-13T16:06:15.340 INFO:tasks.workunit.client.0.smithi112.stdout:) 2024-09-13T16:06:15.375 INFO:tasks.workunit.client.0.smithi112.stdout:--> loop: 4 ~ true / 58 / 52 / false / 2024-09-13T16:06:14.359215+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2024-09-13T16:06:15.375 INFO:tasks.workunit.client.0.smithi112.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: true 2024-09-13T16:06:15.375 INFO:tasks.workunit.client.0.smithi112.stdout:WaitingActive - 'dmp_state_has_scrubbing' actual value (true) matches expected (true) (negation: 0) 2024-09-13T16:06:15.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2024-09-13T16:06:15.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:644: TEST_dump_scrub_schedule: perf_counters td/osd-scrub-test 3 2024-09-13T16:06:15.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-09-13T16:06:15.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2024-09-13T16:06:15.377 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2024-09-13T16:06:15.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2024-09-13T16:06:15.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:06:15.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-09-13T16:06:15.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:06:15.511 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:06:15.511 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:06:15.511 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:06:15.511 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:06:15.511 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:06:15.512 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:06:15.512 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.512 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:06:15.512 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:06:15.512 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:06:15.512 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:06:15.512 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:06:15.512 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.512 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.512 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.512 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.512 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:06:15.512 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:06:15.513 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.513 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.513 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.513 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.513 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:06:15.513 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:06:15.513 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:06:15.513 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:06:15.513 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:06:15.513 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:06:15.513 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:06:15.513 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.513 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.514 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.514 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.514 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:06:15.514 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:06:15.514 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:06:15.514 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:06:15.514 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.514 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.514 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.514 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.514 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:06:15.514 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.515 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.515 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:06:15.515 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:06:15.515 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:06:15.515 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:06:15.515 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:06:15.515 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:06:15.515 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.515 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:06:15.515 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:06:15.515 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:06:15.515 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:06:15.515 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:06:15.515 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.516 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.516 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.516 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.516 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:06:15.516 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:06:15.516 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.516 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.516 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.516 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.517 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:06:15.517 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:06:15.517 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:06:15.517 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:06:15.517 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:06:15.517 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:06:15.517 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:06:15.517 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.517 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.517 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.517 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.517 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:06:15.517 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:06:15.517 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:06:15.518 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:06:15.518 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.518 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.518 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.518 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.518 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:06:15.518 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.518 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.518 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:06:15.518 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:06:15.518 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:06:15.518 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:06:15.519 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:06:15.519 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:06:15.519 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.519 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:06:15.519 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:06:15.519 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:06:15.519 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:06:15.519 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:06:15.519 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.519 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.519 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.519 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.519 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:06:15.519 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:06:15.520 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.520 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.520 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.520 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.520 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:06:15.520 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:06:15.520 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:06:15.520 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:06:15.520 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:06:15.520 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:06:15.520 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:06:15.520 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.521 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.521 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.521 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.521 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:06:15.521 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:06:15.521 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:06:15.521 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:06:15.521 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.521 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.521 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.521 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.521 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:06:15.521 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.522 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.522 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:06:15.522 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:06:15.522 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:06:15.522 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:06:15.522 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:06:15.522 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:06:15.522 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.522 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:06:15.522 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:06:15.522 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:06:15.522 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:06:15.522 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:06:15.523 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.523 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.523 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.523 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.523 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:06:15.523 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:06:15.523 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.523 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.523 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.523 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.523 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:06:15.523 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:06:15.523 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:06:15.523 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:06:15.524 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:06:15.524 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:06:15.524 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:06:15.524 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.524 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.524 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.525 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.525 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:06:15.525 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:06:15.525 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:06:15.525 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:06:15.525 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.525 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.525 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.525 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.525 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:06:15.525 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.525 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.525 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:06:15.525 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:15.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:06:15.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-09-13T16:06:15.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:06:15.655 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:06:15.655 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:06:15.656 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:06:15.656 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:06:15.656 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:06:15.656 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:06:15.656 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.656 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:06:15.656 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:06:15.656 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:06:15.656 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:06:15.656 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:06:15.656 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.656 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.657 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.657 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.657 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:06:15.657 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:06:15.657 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.657 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.657 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.657 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.657 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:06:15.657 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:06:15.657 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:06:15.657 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:06:15.657 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:06:15.658 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:06:15.658 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:06:15.658 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.658 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.658 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.658 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.658 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:06:15.658 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:06:15.658 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:06:15.658 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:06:15.658 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.658 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.658 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.658 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.659 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:06:15.659 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.659 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.659 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:06:15.659 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:06:15.659 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:06:15.659 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:06:15.659 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:06:15.659 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:06:15.659 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.659 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:06:15.659 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 1, 2024-09-13T16:06:15.659 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 1, 2024-09-13T16:06:15.659 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 1, 2024-09-13T16:06:15.660 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:06:15.660 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 1, 2024-09-13T16:06:15.660 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0.115996978, 2024-09-13T16:06:15.660 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0.115996978 2024-09-13T16:06:15.660 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.660 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:06:15.660 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:06:15.660 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.660 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.660 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.660 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.660 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:06:15.660 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 1, 2024-09-13T16:06:15.661 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:06:15.661 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:06:15.661 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:06:15.661 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:06:15.661 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:06:15.661 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.661 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.661 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.661 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.661 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:06:15.661 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:06:15.661 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 1, 2024-09-13T16:06:15.662 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:06:15.662 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.662 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.662 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.662 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.662 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 2 2024-09-13T16:06:15.662 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.662 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.662 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:06:15.662 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:06:15.662 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:06:15.662 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:06:15.662 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:06:15.662 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:06:15.663 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.663 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:06:15.663 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:06:15.663 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:06:15.663 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:06:15.663 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:06:15.663 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.663 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.663 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.663 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.663 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:06:15.663 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:06:15.663 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.663 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.664 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.664 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.664 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:06:15.664 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:06:15.664 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:06:15.664 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:06:15.664 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:06:15.664 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:06:15.664 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:06:15.664 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.664 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.664 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.664 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.664 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:06:15.665 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:06:15.665 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:06:15.665 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:06:15.665 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.665 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.665 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.665 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.665 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:06:15.665 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.665 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.665 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:06:15.665 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:06:15.665 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:06:15.666 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:06:15.666 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:06:15.666 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:06:15.666 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.666 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:06:15.666 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 1, 2024-09-13T16:06:15.666 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 1, 2024-09-13T16:06:15.666 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 1, 2024-09-13T16:06:15.666 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:06:15.666 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 1, 2024-09-13T16:06:15.666 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 2.007947754, 2024-09-13T16:06:15.666 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 2.007947754 2024-09-13T16:06:15.666 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.666 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:06:15.667 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:06:15.667 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.667 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.667 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.667 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.667 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:06:15.667 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 1, 2024-09-13T16:06:15.667 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:06:15.667 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:06:15.667 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:06:15.667 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 1, 2024-09-13T16:06:15.668 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:06:15.668 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 1, 2024-09-13T16:06:15.668 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.668 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.668 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.669 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:06:15.669 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:06:15.669 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:06:15.669 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:06:15.669 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.669 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.669 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.669 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.669 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 2 2024-09-13T16:06:15.669 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.669 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.669 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:06:15.669 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:15.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:06:15.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2024-09-13T16:06:15.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:06:15.796 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:06:15.796 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:06:15.797 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:06:15.797 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:06:15.797 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:06:15.797 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:06:15.797 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.797 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:06:15.797 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:06:15.797 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:06:15.797 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:06:15.797 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:06:15.797 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.797 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.798 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.798 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.798 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:06:15.798 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:06:15.798 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.798 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.798 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.798 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.798 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:06:15.798 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:06:15.798 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:06:15.798 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:06:15.798 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:06:15.799 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:06:15.799 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:06:15.799 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.799 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.799 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.799 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.799 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:06:15.799 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:06:15.799 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:06:15.799 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:06:15.799 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.800 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.800 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.800 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.800 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:06:15.800 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.800 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.800 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:06:15.800 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:06:15.800 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:06:15.800 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:06:15.800 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:06:15.800 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:06:15.801 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.801 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:06:15.801 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:06:15.801 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:06:15.801 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:06:15.801 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:06:15.801 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.801 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.801 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.801 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.801 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:06:15.802 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:06:15.802 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.802 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.802 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.802 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.802 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:06:15.802 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:06:15.802 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:06:15.802 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:06:15.802 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:06:15.802 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:06:15.802 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:06:15.802 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.802 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.803 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.803 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.803 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:06:15.803 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:06:15.803 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:06:15.803 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:06:15.803 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.803 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.803 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.803 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.803 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:06:15.803 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.803 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.803 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:06:15.803 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:06:15.804 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:06:15.804 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:06:15.804 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:06:15.804 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:06:15.804 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.804 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:06:15.804 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:06:15.804 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:06:15.804 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:06:15.804 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:06:15.804 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.804 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.804 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.805 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.805 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:06:15.805 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:06:15.805 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.805 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.805 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.805 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.805 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:06:15.805 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:06:15.806 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:06:15.806 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:06:15.806 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:06:15.806 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:06:15.806 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:06:15.806 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.806 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.806 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.806 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.806 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:06:15.806 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:06:15.806 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:06:15.807 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:06:15.807 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.807 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.807 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.807 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.807 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:06:15.807 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.807 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.807 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:06:15.807 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:06:15.807 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:06:15.807 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:06:15.807 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:06:15.807 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:06:15.808 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.808 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:06:15.808 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:06:15.808 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:06:15.808 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:06:15.808 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:06:15.808 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.808 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.808 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.808 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.808 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:06:15.809 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:06:15.809 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.809 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.809 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.809 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.809 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:06:15.809 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:06:15.809 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:06:15.809 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:06:15.809 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:06:15.809 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:06:15.809 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:06:15.810 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.810 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.810 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.811 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.811 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:06:15.811 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:06:15.811 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:06:15.811 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:06:15.811 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:06:15.811 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:06:15.811 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:06:15.811 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:06:15.811 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:06:15.811 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.811 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:06:15.811 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:06:15.812 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:06:15.812 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Teardown Test TEST_dump_scrub_schedule ------------------ 2024-09-13T16:06:15.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_dump_scrub_schedule ------------------' 2024-09-13T16:06:15.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-09-13T16:06:15.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:06:15.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:06:15.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:06:15.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:06:15.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:06:15.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:06:15.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:06:15.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:06:16.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:06:16.121 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:06:16.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:06:16.123 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:06:16.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:06:16.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:06:16.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:06:16.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:06:16.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:06:16.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:06:16.127 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:06:16.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:06:16.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:06:16.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:06:16.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:06:16.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:06:16.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:06:16.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:06:16.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:06:16.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:06:16.141 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Complete Test TEST_dump_scrub_schedule ------------------ 2024-09-13T16:06:16.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_dump_scrub_schedule ------------------' 2024-09-13T16:06:16.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-13T16:06:16.142 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Prepare Test TEST_interval_changes ------------------- 2024-09-13T16:06:16.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_interval_changes -------------------' 2024-09-13T16:06:16.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-09-13T16:06:16.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2024-09-13T16:06:16.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2024-09-13T16:06:16.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:06:16.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:06:16.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:06:16.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:06:16.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:06:16.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:06:16.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:06:16.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:06:16.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:06:16.145 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:06:16.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:06:16.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:06:16.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:06:16.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:06:16.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:06:16.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:06:16.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:06:16.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:06:16.150 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:06:16.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:06:16.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:06:16.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:06:16.154 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:06:16.154 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:06:16.154 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:06:16.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:06:16.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:06:16.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:06:16.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2024-09-13T16:06:16.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T16:06:16.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:06:16.159 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:06:16.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.528094 2024-09-13T16:06:16.160 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T16:06:16.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T16:06:16.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T16:06:16.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-13T16:06:16.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_interval_changes -----------------------' 2024-09-13T16:06:16.161 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Run Test TEST_interval_changes ----------------------- 2024-09-13T16:06:16.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_interval_changes td/osd-scrub-test 2024-09-13T16:06:16.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:149: TEST_interval_changes: local poolname=test 2024-09-13T16:06:16.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:150: TEST_interval_changes: local OSDS=2 2024-09-13T16:06:16.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:151: TEST_interval_changes: local objects=10 2024-09-13T16:06:16.163 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:153: TEST_interval_changes: expr 24 '*' 60 '*' 60 2024-09-13T16:06:16.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:153: TEST_interval_changes: local day=86400 2024-09-13T16:06:16.163 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:154: TEST_interval_changes: expr 86400 '*' 7 2024-09-13T16:06:16.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:154: TEST_interval_changes: local week=604800 2024-09-13T16:06:16.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: local min_interval=86400 2024-09-13T16:06:16.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:156: TEST_interval_changes: local max_interval=604800 2024-09-13T16:06:16.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: local WAIT_FOR_UPDATE=15 2024-09-13T16:06:16.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:159: TEST_interval_changes: TESTDATA=testdata.528094 2024-09-13T16:06:16.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:162: TEST_interval_changes: run_mon td/osd-scrub-test a --osd_pool_default_size=2 2024-09-13T16:06:16.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2024-09-13T16:06:16.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T16:06:16.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T16:06:16.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T16:06:16.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2024-09-13T16:06:16.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=2 2024-09-13T16:06:16.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T16:06:16.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:06:16.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:06:16.688 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:06:16.688 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:06:16.688 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:06:16.688 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:06:16.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-13T16:06:16.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T16:06:16.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T16:06:16.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T16:06:16.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T16:06:16.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T16:06:16.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T16:06:16.934 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T16:06:16.934 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T16:06:16.934 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T16:06:16.935 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:06:16.935 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:06:16.935 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:06:16.935 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-mon.a.asok 2024-09-13T16:06:16.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T16:06:16.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.528094/ceph-mon.a.asok config get fsid 2024-09-13T16:06:17.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T16:06:17.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T16:06:17.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T16:06:17.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T16:06:17.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T16:06:17.054 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T16:06:17.054 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T16:06:17.054 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T16:06:17.054 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:06:17.055 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:06:17.055 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:06:17.055 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-mon.a.asok 2024-09-13T16:06:17.055 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T16:06:17.055 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.528094/ceph-mon.a.asok config get mon_host 2024-09-13T16:06:17.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:163: TEST_interval_changes: run_mgr td/osd-scrub-test x 2024-09-13T16:06:17.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2024-09-13T16:06:17.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T16:06:17.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T16:06:17.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T16:06:17.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2024-09-13T16:06:17.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T16:06:18.098 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T16:06:18.098 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:06:18.098 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:06:18.098 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:06:18.098 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:06:18.098 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:06:18.098 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:06:18.099 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T16:06:18.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T16:06:18.133 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: expr 2 - 1 2024-09-13T16:06:18.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: seq 0 1 2024-09-13T16:06:18.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:06:18.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:166: TEST_interval_changes: run_osd td/osd-scrub-test 0 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-09-13T16:06:18.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:06:18.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:06:18.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T16:06:18.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:06:18.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-13T16:06:18.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:06:18.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:06:18.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:06:18.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:06:18.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-13T16:06:18.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-13T16:06:18.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:06:18.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:06:18.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:06:18.141 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:06:18.141 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:06:18.141 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:06:18.141 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:06:18.141 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:06:18.141 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:06:18.141 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:06:18.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:06:18.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:06:18.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:06:18.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:06:18.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:06:18.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:06:18.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:06:18.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:06:18.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:06:18.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:06:18.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:06:18.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2024-09-13T16:06:18.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-13T16:06:18.142 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:06:18.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=51f228c5-1654-4024-85e0-b9127a7d3cbe 2024-09-13T16:06:18.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 51f228c5-1654-4024-85e0-b9127a7d3cbe' 2024-09-13T16:06:18.143 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 51f228c5-1654-4024-85e0-b9127a7d3cbe 2024-09-13T16:06:18.143 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:06:18.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD6YuRmNr1KCRAACi1Yi1gYtih+INaoN6faXg== 2024-09-13T16:06:18.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD6YuRmNr1KCRAACi1Yi1gYtih+INaoN6faXg=="}' 2024-09-13T16:06:18.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 51f228c5-1654-4024-85e0-b9127a7d3cbe -i td/osd-scrub-test/0/new.json 2024-09-13T16:06:18.459 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:06:18.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-13T16:06:18.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQD6YuRmNr1KCRAACi1Yi1gYtih+INaoN6faXg== --osd-uuid 51f228c5-1654-4024-85e0-b9127a7d3cbe 2024-09-13T16:06:18.508 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:06:18.505+0000 7f81624277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:06:18.569 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:06:18.565+0000 7f81624277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:06:18.571 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:06:18.569+0000 7f81624277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:06:18.571 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:06:18.569+0000 7f81624277c0 -1 bdev(0x55ff1bd5c800 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2024-09-13T16:06:18.571 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:06:18.569+0000 7f81624277c0 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-13T16:06:26.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-13T16:06:26.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:06:26.665 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T16:06:26.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T16:06:26.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:06:27.072 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T16:06:27.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T16:06:27.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:06:27.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:06:27.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:06:27.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-09-13T16:06:27.106 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:06:27.101+0000 7f93d81b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:06:27.174 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:06:27.169+0000 7f93d81b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:06:27.175 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:06:27.173+0000 7f93d81b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:06:27.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T16:06:27.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:06:27.396 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:06:27.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T16:06:27.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:06:27.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:06:27.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:27.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:06:27.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:27.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:27.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:28.658 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:06:28.653+0000 7f93d81b97c0 -1 Falling back to public interface 2024-09-13T16:06:28.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:28.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:28.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:06:28.717 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:06:28.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:28.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:29.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:30.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:30.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:30.048 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:06:30.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:06:30.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:30.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:30.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:31.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:31.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:31.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:06:31.382 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:06:31.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:31.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:31.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:31.885 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:06:31.881+0000 7f93d81b97c0 -1 osd.0 0 log_to_monitors true 2024-09-13T16:06:32.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:32.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:32.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:06:32.711 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:06:32.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:32.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:33.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:34.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:34.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:34.150 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:06:34.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:06:34.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:34.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:34.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:35.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:35.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:35.488 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:06:35.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T16:06:35.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:35.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:35.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:36.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:36.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:36.821 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:06:36.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T16:06:36.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:36.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:37.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:38.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:38.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:38.275 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:06:38.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T16:06:38.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:38.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:38.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:39.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:39.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:39.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T16:06:39.601 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:06:39.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:39.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:40.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:41.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:41.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:41.144 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:06:41.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T16:06:41.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:41.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:41.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:42.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:42.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:42.487 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:06:42.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T16:06:42.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:42.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:42.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:43.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:43.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:43.846 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:06:43.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T16:06:43.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:43.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:44.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:45.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:45.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:45.182 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:06:45.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T16:06:45.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:45.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:45.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:46.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:46.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:46.692 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:06:46.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T16:06:46.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:46.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:47.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:48.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:48.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:48.218 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:06:48.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T16:06:48.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:48.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:48.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:49.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:49.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:49.550 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:06:49.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T16:06:49.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:49.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:49.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:50.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:50.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:50.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T16:06:50.999 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:06:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:50.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:51.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:52.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:52.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:52.955 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:06:52.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T16:06:52.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:52.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:53.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:54.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:54.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:54.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T16:06:54.748 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:06:54.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:54.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:55.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:56.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:56.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:56.511 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:06:56.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T16:06:56.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:56.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:57.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:06:58.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:06:58.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:06:58.192 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:06:58.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T16:06:58.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:06:58.193 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:06:59.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:00.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:00.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:00.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T16:07:00.204 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:07:00.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:00.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:07:01.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:02.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:02.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:02.163 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:07:02.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T16:07:02.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:02.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:07:02.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:03.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:03.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:03.853 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:07:03.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T16:07:03.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:03.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:07:04.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:05.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:05.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:05.334 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T16:07:05.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T16:07:05.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:05.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:07:05.653 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2113269669,v1:127.0.0.1:6803/2113269669] [v2:127.0.0.1:6804/2113269669,v1:127.0.0.1:6805/2113269669] exists,up 51f228c5-1654-4024-85e0-b9127a7d3cbe 2024-09-13T16:07:05.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:07:05.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:07:05.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:07:05.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:07:05.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:166: TEST_interval_changes: run_osd td/osd-scrub-test 1 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-09-13T16:07:05.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:07:05.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:07:05.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T16:07:05.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:07:05.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-13T16:07:05.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:07:05.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:07:05.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:07:05.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:07:05.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-13T16:07:05.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-13T16:07:05.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:07:05.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:07:05.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:07:05.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:07:05.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:07:05.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:07:05.659 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:07:05.659 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:07:05.659 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:07:05.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:07:05.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:07:05.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:07:05.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:07:05.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:07:05.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:07:05.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:07:05.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:07:05.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:07:05.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:07:05.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:07:05.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:07:05.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2024-09-13T16:07:05.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-13T16:07:05.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:07:05.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ff04cba8-2c35-4fa0-a29b-0989480517fe 2024-09-13T16:07:05.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ff04cba8-2c35-4fa0-a29b-0989480517fe' 2024-09-13T16:07:05.661 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 ff04cba8-2c35-4fa0-a29b-0989480517fe 2024-09-13T16:07:05.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:07:05.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQApY+Rm9WM1KBAAVUmhGJOzZHI66Wcjn52BVw== 2024-09-13T16:07:05.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQApY+Rm9WM1KBAAVUmhGJOzZHI66Wcjn52BVw=="}' 2024-09-13T16:07:05.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ff04cba8-2c35-4fa0-a29b-0989480517fe -i td/osd-scrub-test/1/new.json 2024-09-13T16:07:06.218 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:07:06.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-13T16:07:06.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQApY+Rm9WM1KBAAVUmhGJOzZHI66Wcjn52BVw== --osd-uuid ff04cba8-2c35-4fa0-a29b-0989480517fe 2024-09-13T16:07:06.273 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:07:06.268+0000 7fa4d9e8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:07:06.354 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:07:06.352+0000 7fa4d9e8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:07:06.356 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:07:06.352+0000 7fa4d9e8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:07:06.356 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:07:06.352+0000 7fa4d9e8b7c0 -1 bdev(0x5605d26afc00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2024-09-13T16:07:06.357 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:07:06.352+0000 7fa4d9e8b7c0 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-13T16:07:14.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-13T16:07:14.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:07:14.697 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T16:07:14.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T16:07:14.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:07:15.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T16:07:15.099 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T16:07:15.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-09-13T16:07:15.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:07:15.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:07:15.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:07:15.131 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:07:15.132+0000 7fe4de0f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:07:15.133 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:07:15.132+0000 7fe4de0f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:07:15.135 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:07:15.132+0000 7fe4de0f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:07:15.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T16:07:15.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:07:15.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T16:07:15.412 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:07:15.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:07:15.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:07:15.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:15.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:07:15.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:15.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:15.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:16.674 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:07:16.672+0000 7fe4de0f77c0 -1 Falling back to public interface 2024-09-13T16:07:16.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:16.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:16.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:07:16.765 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:07:16.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:16.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:17.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:18.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:18.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:18.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:07:18.117 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:07:18.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:18.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:18.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:19.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:19.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:19.449 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:07:19.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:07:19.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:19.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:19.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:19.839 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:07:19.840+0000 7fe4de0f77c0 -1 osd.1 0 log_to_monitors true 2024-09-13T16:07:20.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:20.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:20.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:07:20.771 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:07:20.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:20.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:21.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:22.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:22.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:22.183 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:07:22.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:07:22.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:22.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:22.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:23.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:23.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:23.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T16:07:23.653 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:07:23.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:23.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:24.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:25.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:25.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:25.150 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:07:25.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T16:07:25.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:25.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:25.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:26.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:26.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:26.487 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:07:26.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T16:07:26.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:26.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:26.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:27.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:27.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:27.908 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:07:27.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T16:07:27.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:27.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:28.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:29.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:29.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:29.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T16:07:29.336 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:07:29.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:29.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:29.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:30.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:30.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:30.808 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:07:30.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T16:07:30.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:30.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:31.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:32.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:32.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:32.297 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:07:32.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T16:07:32.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:32.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:32.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:33.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:33.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:33.635 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:07:33.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T16:07:33.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:33.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:34.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:35.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:35.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:35.177 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:07:35.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T16:07:35.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:35.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:35.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:36.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:36.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:36.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T16:07:36.538 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:07:36.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:36.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:36.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:37.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:37.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:37.890 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:07:37.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T16:07:37.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:37.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:38.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:39.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:39.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:39.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T16:07:39.223 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:07:39.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:39.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:39.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:40.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:40.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:40.673 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:07:40.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T16:07:40.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:40.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:41.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:42.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:42.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:42.036 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:07:42.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T16:07:42.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:42.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:42.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:43.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:43.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:43.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T16:07:43.712 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:07:43.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:43.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:44.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:45.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:45.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:45.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T16:07:45.627 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:07:45.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:45.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:46.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:47.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:47.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:47.646 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:07:47.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T16:07:47.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:47.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:48.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:49.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:49.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:49.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T16:07:49.818 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:07:49.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:49.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:50.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:51.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:51.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:51.561 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:07:51.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T16:07:51.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:51.561 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:52.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:07:53.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:07:53.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:07:53.338 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T16:07:53.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T16:07:53.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:07:53.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:07:53.815 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2640307867,v1:127.0.0.1:6811/2640307867] [v2:127.0.0.1:6812/2640307867,v1:127.0.0.1:6813/2640307867] exists,up ff04cba8-2c35-4fa0-a29b-0989480517fe 2024-09-13T16:07:53.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:07:53.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:07:53.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:07:53.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: create_pool test 1 1 2024-09-13T16:07:53.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-13T16:07:54.353 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'test' created 2024-09-13T16:07:54.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T16:07:55.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:171: TEST_interval_changes: wait_for_clean 2024-09-13T16:07:55.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:07:55.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:07:55.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:07:55.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:07:55.377 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:07:55.377 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:07:55.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:07:55.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:07:55.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:07:55.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:07:55.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:07:55.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:07:55.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:07:55.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:07:55.502 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:07:55.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:07:55.832 INFO:tasks.workunit.client.0.smithi112.stderr:1' 2024-09-13T16:07:55.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:07:55.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:07:55.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:07:55.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836495 2024-09-13T16:07:55.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836495 2024-09-13T16:07:55.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836495' 2024-09-13T16:07:55.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:07:55.977 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:07:56.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672969 2024-09-13T16:07:56.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672969 2024-09-13T16:07:56.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836495 1-42949672969' 2024-09-13T16:07:56.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:07:56.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836495 2024-09-13T16:07:56.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:07:56.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:07:56.223 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836495 2024-09-13T16:07:56.224 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:07:56.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836495 2024-09-13T16:07:56.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-09-13T16:07:56.225 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836495 2024-09-13T16:07:56.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:07:56.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836493 -lt 21474836495 2024-09-13T16:07:56.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:07:57.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:07:57.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:07:57.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836495 -lt 21474836495 2024-09-13T16:07:57.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:07:57.976 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672969 2024-09-13T16:07:57.976 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:07:57.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:07:57.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672969 2024-09-13T16:07:57.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:07:57.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672969 2024-09-13T16:07:57.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-13T16:07:57.980 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672969 2024-09-13T16:07:57.980 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:07:58.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672969 -lt 42949672969 2024-09-13T16:07:58.370 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:07:58.370 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:07:58.371 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:07:58.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:07:58.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:07:58.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:07:58.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:07:58.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:07:58.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:07:58.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:07:58.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:07:59.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T16:07:59.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:07:59.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:07:59.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:07:59.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T16:07:59.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T16:07:59.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T16:07:59.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T16:07:59.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T16:07:59.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T16:07:59.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T16:07:59.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:07:59.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:07:59.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:07:59.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:07:59.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:07:59.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:07:59.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:07:59.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T16:07:59.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:07:59.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:07:59.924 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:08:00.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T16:08:00.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T16:08:00.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T16:08:00.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T16:08:00.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T16:08:00.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T16:08:00.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T16:08:00.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T16:08:00.290 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T16:08:00.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T16:08:00.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T16:08:00.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T16:08:00.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T16:08:00.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T16:08:00.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T16:08:00.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:08:00.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:08:00.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:08:00.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:08:00.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:08:00.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:08:00.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:08:01.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:08:01.219 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:08:01.219 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:08:01.219 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:08:01.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:08:01.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:08:01.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:08:01.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: ceph osd dump 2024-09-13T16:08:01.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: grep '^pool.*['\'']test['\'']' 2024-09-13T16:08:01.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: awk '{ print $2 }' 2024-09-13T16:08:01.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: local poolid=1 2024-09-13T16:08:01.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:174: TEST_interval_changes: dd if=/dev/urandom of=testdata.528094 bs=1032 count=1 2024-09-13T16:08:01.895 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T16:08:01.895 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T16:08:01.895 INFO:tasks.workunit.client.0.smithi112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000109172 s, 9.5 MB/s 2024-09-13T16:08:01.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: seq 1 10 2024-09-13T16:08:01.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-13T16:08:01.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj1 testdata.528094 2024-09-13T16:08:02.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-13T16:08:02.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj2 testdata.528094 2024-09-13T16:08:02.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-13T16:08:02.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj3 testdata.528094 2024-09-13T16:08:02.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-13T16:08:02.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj4 testdata.528094 2024-09-13T16:08:02.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-13T16:08:02.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj5 testdata.528094 2024-09-13T16:08:02.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-13T16:08:02.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj6 testdata.528094 2024-09-13T16:08:03.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-13T16:08:03.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj7 testdata.528094 2024-09-13T16:08:03.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-13T16:08:03.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj8 testdata.528094 2024-09-13T16:08:03.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-13T16:08:03.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj9 testdata.528094 2024-09-13T16:08:03.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-09-13T16:08:03.846 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj10 testdata.528094 2024-09-13T16:08:04.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:179: TEST_interval_changes: rm -f testdata.528094 2024-09-13T16:08:04.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:181: TEST_interval_changes: get_primary test obj1 2024-09-13T16:08:04.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-13T16:08:04.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T16:08:04.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T16:08:04.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-13T16:08:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:181: TEST_interval_changes: local primary=1 2024-09-13T16:08:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:184: TEST_interval_changes: check_dump_scrubs 1 '1 day' '1 week' 2024-09-13T16:08:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-09-13T16:08:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=1 day' 2024-09-13T16:08:04.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=1 week' 2024-09-13T16:08:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-09-13T16:08:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T16:08:04.390 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T16:08:04.391 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:08:04.391 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:08:04.391 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:08:04.391 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-osd.1.asok 2024-09-13T16:08:04.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-09-13T16:08:04.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.528094/ceph-osd.1.asok dump_scrubs 2024-09-13T16:08:04.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-09-13T16:08:04.504 INFO:tasks.workunit.client.0.smithi112.stderr: { 2024-09-13T16:08:04.504 INFO:tasks.workunit.client.0.smithi112.stderr: "pgid": "1.0", 2024-09-13T16:08:04.504 INFO:tasks.workunit.client.0.smithi112.stderr: "sched_time": "2024-09-14T16:07:54.186784+0000", 2024-09-13T16:08:04.505 INFO:tasks.workunit.client.0.smithi112.stderr: "orig_sched_time": "2024-09-14T16:07:54.186784+0000", 2024-09-13T16:08:04.505 INFO:tasks.workunit.client.0.smithi112.stderr: "deadline": "2024-09-20T16:07:54.186784+0000", 2024-09-13T16:08:04.505 INFO:tasks.workunit.client.0.smithi112.stderr: "forced": false 2024-09-13T16:08:04.505 INFO:tasks.workunit.client.0.smithi112.stderr: }, 2024-09-13T16:08:04.505 INFO:tasks.workunit.client.0.smithi112.stderr: { 2024-09-13T16:08:04.505 INFO:tasks.workunit.client.0.smithi112.stderr: "pgid": "1.0", 2024-09-13T16:08:04.506 INFO:tasks.workunit.client.0.smithi112.stderr: "sched_time": "2024-09-20T16:07:54.186784+0000", 2024-09-13T16:08:04.506 INFO:tasks.workunit.client.0.smithi112.stderr: "orig_sched_time": "2024-09-20T16:07:54.186784+0000", 2024-09-13T16:08:04.506 INFO:tasks.workunit.client.0.smithi112.stderr: "deadline": "2024-09-20T16:07:54.186784+0000", 2024-09-13T16:08:04.506 INFO:tasks.workunit.client.0.smithi112.stderr: "forced": false 2024-09-13T16:08:04.506 INFO:tasks.workunit.client.0.smithi112.stderr: } 2024-09-13T16:08:04.506 INFO:tasks.workunit.client.0.smithi112.stderr:]' 2024-09-13T16:08:04.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-14T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-14T16:07:54.186784+0000",' '"deadline":' '"2024-09-20T16:07:54.186784+0000",' '"forced":' false '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"deadline":' '"2024-09-20T16:07:54.186784+0000",' '"forced":' false '}' ']' 2024-09-13T16:08:04.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-09-13T16:08:04.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-09-14T16:07:54.186784+0000"' 2024-09-13T16:08:04.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-09-14T16:07:54.186784+0000 2024-09-13T16:08:04.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-13T16:08:04.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-09-14T16:07:54.186784+0000 2024-09-13T16:08:04.541 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 day' 2024-09-13T16:08:04.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-09-14 = 2024-09-14 2024-09-13T16:08:04.542 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-14T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-14T16:07:54.186784+0000",' '"deadline":' '"2024-09-20T16:07:54.186784+0000",' '"forced":' false '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"deadline":' '"2024-09-20T16:07:54.186784+0000",' '"forced":' false '}' ']' 2024-09-13T16:08:04.542 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-09-13T16:08:04.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-09-20T16:07:54.186784+0000"' 2024-09-13T16:08:04.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-09-20T16:07:54.186784+0000 2024-09-13T16:08:04.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-09-20T16:07:54.186784+0000 2024-09-13T16:08:04.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-13T16:08:04.581 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 week' 2024-09-13T16:08:04.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-09-20 = 2024-09-20 2024-09-13T16:08:04.584 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: get_asok_path osd.1 2024-09-13T16:08:04.584 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T16:08:04.584 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T16:08:04.584 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:08:04.584 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:08:04.584 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:08:04.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-osd.1.asok 2024-09-13T16:08:04.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: expr 86400 '*' 2 2024-09-13T16:08:04.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: CEPH_ARGS= 2024-09-13T16:08:04.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.528094/ceph-osd.1.asok config set osd_scrub_min_interval 172800 2024-09-13T16:08:04.691 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:08:04.709 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_scrub_min_interval = '' " 2024-09-13T16:08:04.709 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:08:04.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:188: TEST_interval_changes: sleep 15 2024-09-13T16:08:19.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:189: TEST_interval_changes: check_dump_scrubs 1 '2 days' '1 week' 2024-09-13T16:08:19.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-09-13T16:08:19.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=2 days' 2024-09-13T16:08:19.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=1 week' 2024-09-13T16:08:19.708 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-09-13T16:08:19.708 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T16:08:19.708 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T16:08:19.709 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:08:19.709 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:08:19.709 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:08:19.709 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-osd.1.asok 2024-09-13T16:08:19.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-09-13T16:08:19.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.528094/ceph-osd.1.asok dump_scrubs 2024-09-13T16:08:19.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-09-13T16:08:19.838 INFO:tasks.workunit.client.0.smithi112.stderr: { 2024-09-13T16:08:19.838 INFO:tasks.workunit.client.0.smithi112.stderr: "pgid": "1.0", 2024-09-13T16:08:19.838 INFO:tasks.workunit.client.0.smithi112.stderr: "sched_time": "2024-09-15T16:07:54.186784+0000", 2024-09-13T16:08:19.838 INFO:tasks.workunit.client.0.smithi112.stderr: "orig_sched_time": "2024-09-15T16:07:54.186784+0000", 2024-09-13T16:08:19.839 INFO:tasks.workunit.client.0.smithi112.stderr: "deadline": "2024-09-20T16:07:54.186784+0000", 2024-09-13T16:08:19.839 INFO:tasks.workunit.client.0.smithi112.stderr: "forced": false 2024-09-13T16:08:19.839 INFO:tasks.workunit.client.0.smithi112.stderr: }, 2024-09-13T16:08:19.839 INFO:tasks.workunit.client.0.smithi112.stderr: { 2024-09-13T16:08:19.839 INFO:tasks.workunit.client.0.smithi112.stderr: "pgid": "1.0", 2024-09-13T16:08:19.839 INFO:tasks.workunit.client.0.smithi112.stderr: "sched_time": "2024-09-20T16:07:54.186784+0000", 2024-09-13T16:08:19.839 INFO:tasks.workunit.client.0.smithi112.stderr: "orig_sched_time": "2024-09-20T16:07:54.186784+0000", 2024-09-13T16:08:19.839 INFO:tasks.workunit.client.0.smithi112.stderr: "deadline": "2024-09-20T16:07:54.186784+0000", 2024-09-13T16:08:19.839 INFO:tasks.workunit.client.0.smithi112.stderr: "forced": false 2024-09-13T16:08:19.839 INFO:tasks.workunit.client.0.smithi112.stderr: } 2024-09-13T16:08:19.839 INFO:tasks.workunit.client.0.smithi112.stderr:]' 2024-09-13T16:08:19.840 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-09-13T16:08:19.840 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-15T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-15T16:07:54.186784+0000",' '"deadline":' '"2024-09-20T16:07:54.186784+0000",' '"forced":' false '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"deadline":' '"2024-09-20T16:07:54.186784+0000",' '"forced":' false '}' ']' 2024-09-13T16:08:19.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-09-15T16:07:54.186784+0000"' 2024-09-13T16:08:19.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-09-15T16:07:54.186784+0000 2024-09-13T16:08:19.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-13T16:08:19.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-09-15T16:07:54.186784+0000 2024-09-13T16:08:19.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2024-09-13T16:08:19.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-09-15 = 2024-09-15 2024-09-13T16:08:19.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-15T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-15T16:07:54.186784+0000",' '"deadline":' '"2024-09-20T16:07:54.186784+0000",' '"forced":' false '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"deadline":' '"2024-09-20T16:07:54.186784+0000",' '"forced":' false '}' ']' 2024-09-13T16:08:19.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-09-13T16:08:19.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-09-20T16:07:54.186784+0000"' 2024-09-13T16:08:19.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-09-20T16:07:54.186784+0000 2024-09-13T16:08:19.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-09-20T16:07:54.186784+0000 2024-09-13T16:08:19.911 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-13T16:08:19.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 week' 2024-09-13T16:08:19.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-09-20 = 2024-09-20 2024-09-13T16:08:19.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: get_asok_path osd.1 2024-09-13T16:08:19.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T16:08:19.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T16:08:19.914 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:08:19.914 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:08:19.914 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:08:19.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-osd.1.asok 2024-09-13T16:08:19.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: expr 604800 '*' 2 2024-09-13T16:08:19.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: CEPH_ARGS= 2024-09-13T16:08:19.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.528094/ceph-osd.1.asok config set osd_scrub_max_interval 1209600 2024-09-13T16:08:20.027 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:08:20.027 INFO:tasks.workunit.client.0.smithi112.stdout: "success": "osd_scrub_max_interval = '' " 2024-09-13T16:08:20.027 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:08:20.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:193: TEST_interval_changes: sleep 15 2024-09-13T16:08:35.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:194: TEST_interval_changes: check_dump_scrubs 1 '2 days' '2 week' 2024-09-13T16:08:35.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-09-13T16:08:35.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=2 days' 2024-09-13T16:08:35.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=2 week' 2024-09-13T16:08:35.044 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-09-13T16:08:35.044 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T16:08:35.044 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T16:08:35.044 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:08:35.044 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:08:35.044 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:08:35.044 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-osd.1.asok 2024-09-13T16:08:35.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-09-13T16:08:35.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.528094/ceph-osd.1.asok dump_scrubs 2024-09-13T16:08:35.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-09-13T16:08:35.173 INFO:tasks.workunit.client.0.smithi112.stderr: { 2024-09-13T16:08:35.173 INFO:tasks.workunit.client.0.smithi112.stderr: "pgid": "1.0", 2024-09-13T16:08:35.173 INFO:tasks.workunit.client.0.smithi112.stderr: "sched_time": "2024-09-15T16:07:54.186784+0000", 2024-09-13T16:08:35.173 INFO:tasks.workunit.client.0.smithi112.stderr: "orig_sched_time": "2024-09-15T16:07:54.186784+0000", 2024-09-13T16:08:35.173 INFO:tasks.workunit.client.0.smithi112.stderr: "deadline": "2024-09-27T16:07:54.186784+0000", 2024-09-13T16:08:35.173 INFO:tasks.workunit.client.0.smithi112.stderr: "forced": false 2024-09-13T16:08:35.174 INFO:tasks.workunit.client.0.smithi112.stderr: }, 2024-09-13T16:08:35.174 INFO:tasks.workunit.client.0.smithi112.stderr: { 2024-09-13T16:08:35.174 INFO:tasks.workunit.client.0.smithi112.stderr: "pgid": "1.0", 2024-09-13T16:08:35.174 INFO:tasks.workunit.client.0.smithi112.stderr: "sched_time": "2024-09-20T16:07:54.186784+0000", 2024-09-13T16:08:35.174 INFO:tasks.workunit.client.0.smithi112.stderr: "orig_sched_time": "2024-09-20T16:07:54.186784+0000", 2024-09-13T16:08:35.174 INFO:tasks.workunit.client.0.smithi112.stderr: "deadline": "2024-09-27T16:07:54.186784+0000", 2024-09-13T16:08:35.174 INFO:tasks.workunit.client.0.smithi112.stderr: "forced": false 2024-09-13T16:08:35.174 INFO:tasks.workunit.client.0.smithi112.stderr: } 2024-09-13T16:08:35.174 INFO:tasks.workunit.client.0.smithi112.stderr:]' 2024-09-13T16:08:35.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-15T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-15T16:07:54.186784+0000",' '"deadline":' '"2024-09-27T16:07:54.186784+0000",' '"forced":' false '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"deadline":' '"2024-09-27T16:07:54.186784+0000",' '"forced":' false '}' ']' 2024-09-13T16:08:35.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-09-13T16:08:35.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-09-15T16:07:54.186784+0000"' 2024-09-13T16:08:35.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-09-15T16:07:54.186784+0000 2024-09-13T16:08:35.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-09-15T16:07:54.186784+0000 2024-09-13T16:08:35.210 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-13T16:08:35.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2024-09-13T16:08:35.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-09-15 = 2024-09-15 2024-09-13T16:08:35.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-09-13T16:08:35.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-15T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-15T16:07:54.186784+0000",' '"deadline":' '"2024-09-27T16:07:54.186784+0000",' '"forced":' false '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"deadline":' '"2024-09-27T16:07:54.186784+0000",' '"forced":' false '}' ']' 2024-09-13T16:08:35.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-09-27T16:07:54.186784+0000"' 2024-09-13T16:08:35.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-09-27T16:07:54.186784+0000 2024-09-13T16:08:35.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-09-27T16:07:54.186784+0000 2024-09-13T16:08:35.244 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-13T16:08:35.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 week' 2024-09-13T16:08:35.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-09-27 = 2024-09-27 2024-09-13T16:08:35.248 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:197: TEST_interval_changes: expr 86400 '*' 3 2024-09-13T16:08:35.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:197: TEST_interval_changes: ceph osd pool set test scrub_min_interval 259200 2024-09-13T16:08:35.703 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 1 scrub_min_interval to 259200 2024-09-13T16:08:35.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:198: TEST_interval_changes: sleep 15 2024-09-13T16:08:50.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:199: TEST_interval_changes: check_dump_scrubs 1 '3 days' '2 week' 2024-09-13T16:08:50.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-09-13T16:08:50.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=3 days' 2024-09-13T16:08:50.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=2 week' 2024-09-13T16:08:50.725 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-09-13T16:08:50.725 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T16:08:50.725 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T16:08:50.726 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:08:50.726 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:08:50.726 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:08:50.727 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-osd.1.asok 2024-09-13T16:08:50.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-09-13T16:08:50.727 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.528094/ceph-osd.1.asok dump_scrubs 2024-09-13T16:08:50.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-09-13T16:08:50.857 INFO:tasks.workunit.client.0.smithi112.stderr: { 2024-09-13T16:08:50.857 INFO:tasks.workunit.client.0.smithi112.stderr: "pgid": "1.0", 2024-09-13T16:08:50.857 INFO:tasks.workunit.client.0.smithi112.stderr: "sched_time": "2024-09-16T16:07:54.186784+0000", 2024-09-13T16:08:50.857 INFO:tasks.workunit.client.0.smithi112.stderr: "orig_sched_time": "2024-09-16T16:07:54.186784+0000", 2024-09-13T16:08:50.857 INFO:tasks.workunit.client.0.smithi112.stderr: "deadline": "2024-09-27T16:07:54.186784+0000", 2024-09-13T16:08:50.857 INFO:tasks.workunit.client.0.smithi112.stderr: "forced": false 2024-09-13T16:08:50.857 INFO:tasks.workunit.client.0.smithi112.stderr: }, 2024-09-13T16:08:50.858 INFO:tasks.workunit.client.0.smithi112.stderr: { 2024-09-13T16:08:50.858 INFO:tasks.workunit.client.0.smithi112.stderr: "pgid": "1.0", 2024-09-13T16:08:50.858 INFO:tasks.workunit.client.0.smithi112.stderr: "sched_time": "2024-09-20T16:07:54.186784+0000", 2024-09-13T16:08:50.858 INFO:tasks.workunit.client.0.smithi112.stderr: "orig_sched_time": "2024-09-20T16:07:54.186784+0000", 2024-09-13T16:08:50.858 INFO:tasks.workunit.client.0.smithi112.stderr: "deadline": "2024-09-27T16:07:54.186784+0000", 2024-09-13T16:08:50.858 INFO:tasks.workunit.client.0.smithi112.stderr: "forced": false 2024-09-13T16:08:50.858 INFO:tasks.workunit.client.0.smithi112.stderr: } 2024-09-13T16:08:50.858 INFO:tasks.workunit.client.0.smithi112.stderr:]' 2024-09-13T16:08:50.858 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-16T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-16T16:07:54.186784+0000",' '"deadline":' '"2024-09-27T16:07:54.186784+0000",' '"forced":' false '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"deadline":' '"2024-09-27T16:07:54.186784+0000",' '"forced":' false '}' ']' 2024-09-13T16:08:50.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-09-13T16:08:50.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-09-16T16:07:54.186784+0000"' 2024-09-13T16:08:50.893 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-09-16T16:07:54.186784+0000 2024-09-13T16:08:50.894 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-09-16T16:07:54.186784+0000 2024-09-13T16:08:50.894 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-13T16:08:50.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2024-09-13T16:08:50.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-09-16 = 2024-09-16 2024-09-13T16:08:50.899 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-16T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-16T16:07:54.186784+0000",' '"deadline":' '"2024-09-27T16:07:54.186784+0000",' '"forced":' false '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"deadline":' '"2024-09-27T16:07:54.186784+0000",' '"forced":' false '}' ']' 2024-09-13T16:08:50.899 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-09-13T16:08:50.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-09-27T16:07:54.186784+0000"' 2024-09-13T16:08:50.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-09-27T16:07:54.186784+0000 2024-09-13T16:08:50.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-09-27T16:07:54.186784+0000 2024-09-13T16:08:50.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-13T16:08:50.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 week' 2024-09-13T16:08:50.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-09-27 = 2024-09-27 2024-09-13T16:08:50.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:202: TEST_interval_changes: expr 604800 '*' 3 2024-09-13T16:08:50.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:202: TEST_interval_changes: ceph osd pool set test scrub_max_interval 1814400 2024-09-13T16:08:51.417 INFO:tasks.workunit.client.0.smithi112.stderr:set pool 1 scrub_max_interval to 1814400 2024-09-13T16:08:51.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:203: TEST_interval_changes: sleep 15 2024-09-13T16:09:06.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:204: TEST_interval_changes: check_dump_scrubs 1 '3 days' '3 week' 2024-09-13T16:09:06.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-09-13T16:09:06.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=3 days' 2024-09-13T16:09:06.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=3 week' 2024-09-13T16:09:06.443 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-09-13T16:09:06.443 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-13T16:09:06.443 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-13T16:09:06.444 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:09:06.444 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:09:06.444 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:09:06.444 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-osd.1.asok 2024-09-13T16:09:06.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-09-13T16:09:06.445 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.528094/ceph-osd.1.asok dump_scrubs 2024-09-13T16:09:06.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-09-13T16:09:06.575 INFO:tasks.workunit.client.0.smithi112.stderr: { 2024-09-13T16:09:06.575 INFO:tasks.workunit.client.0.smithi112.stderr: "pgid": "1.0", 2024-09-13T16:09:06.575 INFO:tasks.workunit.client.0.smithi112.stderr: "sched_time": "2024-09-16T16:07:54.186784+0000", 2024-09-13T16:09:06.576 INFO:tasks.workunit.client.0.smithi112.stderr: "orig_sched_time": "2024-09-16T16:07:54.186784+0000", 2024-09-13T16:09:06.576 INFO:tasks.workunit.client.0.smithi112.stderr: "deadline": "2024-10-04T16:07:54.186784+0000", 2024-09-13T16:09:06.576 INFO:tasks.workunit.client.0.smithi112.stderr: "forced": false 2024-09-13T16:09:06.576 INFO:tasks.workunit.client.0.smithi112.stderr: }, 2024-09-13T16:09:06.576 INFO:tasks.workunit.client.0.smithi112.stderr: { 2024-09-13T16:09:06.576 INFO:tasks.workunit.client.0.smithi112.stderr: "pgid": "1.0", 2024-09-13T16:09:06.576 INFO:tasks.workunit.client.0.smithi112.stderr: "sched_time": "2024-09-20T16:07:54.186784+0000", 2024-09-13T16:09:06.576 INFO:tasks.workunit.client.0.smithi112.stderr: "orig_sched_time": "2024-09-20T16:07:54.186784+0000", 2024-09-13T16:09:06.576 INFO:tasks.workunit.client.0.smithi112.stderr: "deadline": "2024-10-04T16:07:54.186784+0000", 2024-09-13T16:09:06.576 INFO:tasks.workunit.client.0.smithi112.stderr: "forced": false 2024-09-13T16:09:06.576 INFO:tasks.workunit.client.0.smithi112.stderr: } 2024-09-13T16:09:06.576 INFO:tasks.workunit.client.0.smithi112.stderr:]' 2024-09-13T16:09:06.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-16T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-16T16:07:54.186784+0000",' '"deadline":' '"2024-10-04T16:07:54.186784+0000",' '"forced":' false '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"deadline":' '"2024-10-04T16:07:54.186784+0000",' '"forced":' false '}' ']' 2024-09-13T16:09:06.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-09-13T16:09:06.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-09-16T16:07:54.186784+0000"' 2024-09-13T16:09:06.613 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-09-16T16:07:54.186784+0000 2024-09-13T16:09:06.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-09-16T16:07:54.186784+0000 2024-09-13T16:09:06.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-13T16:09:06.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2024-09-13T16:09:06.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-09-16 = 2024-09-16 2024-09-13T16:09:06.618 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-16T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-16T16:07:54.186784+0000",' '"deadline":' '"2024-10-04T16:07:54.186784+0000",' '"forced":' false '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"orig_sched_time":' '"2024-09-20T16:07:54.186784+0000",' '"deadline":' '"2024-10-04T16:07:54.186784+0000",' '"forced":' false '}' ']' 2024-09-13T16:09:06.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-09-13T16:09:06.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-10-04T16:07:54.186784+0000"' 2024-09-13T16:09:06.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-10-04T16:07:54.186784+0000 2024-09-13T16:09:06.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-10-04T16:07:54.186784+0000 2024-09-13T16:09:06.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-09-13T16:09:06.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 week' 2024-09-13T16:09:06.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-10-04 = 2024-10-04 2024-09-13T16:09:06.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:205: TEST_interval_changes: perf_counters td/osd-scrub-test 2 2024-09-13T16:09:06.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-09-13T16:09:06.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=2 2024-09-13T16:09:06.658 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 2 - 1 2024-09-13T16:09:06.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 1 2024-09-13T16:09:06.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:09:06.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-09-13T16:09:06.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:09:06.793 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:09:06.793 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:09:06.793 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:09:06.793 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:09:06.793 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:09:06.793 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:09:06.793 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.793 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:09:06.793 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:09:06.793 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:09:06.794 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:09:06.794 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:09:06.794 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.794 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.794 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.794 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.794 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:09:06.794 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:09:06.794 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.794 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.794 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.794 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.794 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:09:06.794 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:09:06.795 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:09:06.795 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:09:06.795 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:09:06.795 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:09:06.795 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:09:06.795 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.795 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.795 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.795 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.795 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:09:06.795 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:09:06.795 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:09:06.795 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:09:06.796 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.796 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.796 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.796 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.796 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:09:06.796 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.796 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.796 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:09:06.796 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:09:06.796 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:09:06.796 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:09:06.796 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:09:06.796 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:09:06.796 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.797 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:09:06.797 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:09:06.797 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:09:06.797 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:09:06.797 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:09:06.797 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.797 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.797 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.797 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.797 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:09:06.797 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:09:06.797 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.797 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.798 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.798 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.798 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:09:06.798 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:09:06.798 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:09:06.798 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:09:06.798 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:09:06.798 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:09:06.798 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:09:06.799 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.799 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.799 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.799 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.799 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:09:06.799 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:09:06.799 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:09:06.799 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:09:06.799 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.799 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.799 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.800 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.800 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:09:06.800 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.800 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.800 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:09:06.800 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:09:06.800 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:09:06.800 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:09:06.800 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:09:06.800 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:09:06.800 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.800 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:09:06.800 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:09:06.801 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:09:06.801 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:09:06.801 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:09:06.801 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.801 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.801 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.801 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.801 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:09:06.801 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:09:06.802 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.802 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.802 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.802 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.802 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:09:06.802 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:09:06.802 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:09:06.802 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:09:06.802 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:09:06.802 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:09:06.802 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:09:06.802 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.802 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.803 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.803 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.803 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:09:06.803 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:09:06.803 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:09:06.803 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:09:06.803 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.803 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.803 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.803 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.803 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:09:06.804 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.804 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.804 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:09:06.804 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:09:06.804 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:09:06.804 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:09:06.804 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:09:06.804 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:09:06.804 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.804 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:09:06.804 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:09:06.804 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:09:06.804 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:09:06.805 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:09:06.805 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.805 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.805 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.805 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.805 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:09:06.805 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:09:06.805 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.805 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.805 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.805 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.805 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:09:06.805 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:09:06.805 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:09:06.805 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:09:06.806 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:09:06.806 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:09:06.806 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:09:06.806 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.806 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.806 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.806 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.806 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:09:06.807 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:09:06.807 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:09:06.807 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:09:06.807 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.807 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.807 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.807 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.807 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:09:06.807 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.807 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.807 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:09:06.807 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:09:06.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:09:06.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-09-13T16:09:06.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:09:06.970 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:09:06.970 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:09:06.970 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:09:06.970 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:09:06.970 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:09:06.970 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:09:06.970 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.971 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:09:06.971 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:09:06.971 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:09:06.971 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:09:06.971 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:09:06.971 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.971 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.971 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.971 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.971 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:09:06.971 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:09:06.971 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.971 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.972 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.972 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.972 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:09:06.972 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:09:06.972 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:09:06.972 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:09:06.972 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:09:06.972 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:09:06.972 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:09:06.972 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.972 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.972 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.972 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.973 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:09:06.973 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:09:06.973 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:09:06.973 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:09:06.973 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.973 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.973 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.973 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.973 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:09:06.973 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.973 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.973 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:09:06.973 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:09:06.974 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:09:06.974 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:09:06.974 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:09:06.974 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:09:06.974 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.974 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:09:06.974 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:09:06.974 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:09:06.974 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:09:06.974 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:09:06.974 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.974 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.974 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.974 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.975 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:09:06.975 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:09:06.975 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.975 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.975 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.975 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.975 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:09:06.975 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:09:06.975 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:09:06.975 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:09:06.975 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:09:06.975 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:09:06.975 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:09:06.976 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.976 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.976 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.976 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.976 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:09:06.976 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:09:06.976 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:09:06.976 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:09:06.976 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.976 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.976 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.976 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.976 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:09:06.976 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.977 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.977 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:09:06.977 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:09:06.977 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:09:06.977 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:09:06.977 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:09:06.977 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:09:06.977 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.977 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:09:06.977 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:09:06.977 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:09:06.977 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:09:06.977 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:09:06.977 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.978 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.978 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.978 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.978 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:09:06.978 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:09:06.978 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.978 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.978 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.978 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.978 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:09:06.978 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:09:06.978 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:09:06.978 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:09:06.979 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:09:06.979 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:09:06.979 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:09:06.979 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.979 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.979 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.979 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.979 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:09:06.979 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:09:06.979 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:09:06.979 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:09:06.979 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.979 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.979 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.980 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.980 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:09:06.980 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.980 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.980 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:09:06.980 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:09:06.980 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:09:06.980 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:09:06.980 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:09:06.980 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:09:06.980 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.980 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:09:06.980 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:09:06.980 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:09:06.981 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:09:06.981 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:09:06.981 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.981 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.981 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.981 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.981 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:09:06.981 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:09:06.981 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.981 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.981 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.981 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.982 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:09:06.982 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:09:06.982 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:09:06.982 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:09:06.982 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:09:06.982 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:09:06.982 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:09:06.983 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.983 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.983 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.983 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.983 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:09:06.983 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:09:06.984 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:09:06.984 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:09:06.984 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:09:06.984 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:09:06.984 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:09:06.984 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:09:06.984 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:09:06.984 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.984 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:09:06.984 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:09:06.984 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:09:06.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_interval_changes ------------------' 2024-09-13T16:09:06.985 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Teardown Test TEST_interval_changes ------------------ 2024-09-13T16:09:06.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-09-13T16:09:06.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:09:06.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:09:06.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:09:06.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:09:06.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:09:06.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:09:06.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:09:06.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:09:07.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:09:07.295 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:09:07.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:09:07.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:09:07.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:09:07.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:09:07.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:09:07.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:09:07.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:09:07.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:09:07.303 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:09:07.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:09:07.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:09:07.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:09:07.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:09:07.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:09:07.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:09:07.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:09:07.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:09:07.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:09:07.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_interval_changes ------------------' 2024-09-13T16:09:07.323 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Complete Test TEST_interval_changes ------------------ 2024-09-13T16:09:07.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-13T16:09:07.324 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Prepare Test TEST_just_deep_scrubs ------------------- 2024-09-13T16:09:07.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_just_deep_scrubs -------------------' 2024-09-13T16:09:07.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-09-13T16:09:07.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2024-09-13T16:09:07.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2024-09-13T16:09:07.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:09:07.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:09:07.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:09:07.325 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:09:07.325 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:09:07.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:09:07.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:09:07.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:09:07.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:09:07.327 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:09:07.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:09:07.328 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:09:07.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:09:07.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:09:07.330 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:09:07.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:09:07.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:09:07.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:09:07.331 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:09:07.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:09:07.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:09:07.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:09:07.335 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:09:07.335 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:09:07.335 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:09:07.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:09:07.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:09:07.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:09:07.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2024-09-13T16:09:07.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T16:09:07.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:09:07.338 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:09:07.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.528094 2024-09-13T16:09:07.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T16:09:07.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T16:09:07.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T16:09:07.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-13T16:09:07.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_just_deep_scrubs -----------------------' 2024-09-13T16:09:07.341 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Run Test TEST_just_deep_scrubs ----------------------- 2024-09-13T16:09:07.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_just_deep_scrubs td/osd-scrub-test 2024-09-13T16:09:07.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:469: TEST_just_deep_scrubs: local dir=td/osd-scrub-test 2024-09-13T16:09:07.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:470: TEST_just_deep_scrubs: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-09-13T16:09:07.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:470: TEST_just_deep_scrubs: local -A cluster_conf 2024-09-13T16:09:07.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:476: TEST_just_deep_scrubs: standard_scrub_cluster td/osd-scrub-test cluster_conf 2024-09-13T16:09:07.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-test 2024-09-13T16:09:07.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2024-09-13T16:09:07.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2024-09-13T16:09:07.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2024-09-13T16:09:07.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:234: standard_scrub_cluster: local poolname=test 2024-09-13T16:09:07.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: args['pool_name']=test 2024-09-13T16:09:07.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: local extra_pars= 2024-09-13T16:09:07.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:237: standard_scrub_cluster: local debug_msg=dbg 2024-09-13T16:09:07.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:240: standard_scrub_cluster: local saved_echo_flag=x 2024-09-13T16:09:07.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: set +x 2024-09-13T16:09:09.364 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 c19d82ab-2b6b-4af8-9029-b5038201c89c 2024-09-13T16:09:09.689 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:09:09.738 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:09:09.733+0000 7f3d8491d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:09:09.774 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:09:09.769+0000 7f3d8491d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:09:09.775 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:09:09.773+0000 7f3d8491d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:09:09.775 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:09:09.773+0000 7f3d8491d7c0 -1 bdev(0x557b36246800 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2024-09-13T16:09:09.776 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:09:09.773+0000 7f3d8491d7c0 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-13T16:09:18.097 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T16:09:18.499 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T16:09:18.536 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:09:18.533+0000 7fe2585f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:09:18.579 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:09:18.577+0000 7fe2585f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:09:18.580 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:09:18.577+0000 7fe2585f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:09:18.799 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:09:19.653 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:09:19.649+0000 7fe2585f97c0 -1 Falling back to public interface 2024-09-13T16:09:20.110 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:09:21.429 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:09:22.727 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:09:22.721+0000 7fe2585f97c0 -1 osd.0 0 log_to_monitors true 2024-09-13T16:09:22.753 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:09:24.137 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:09:25.462 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:09:26.847 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:09:28.165 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:09:29.609 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:09:30.932 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:09:32.275 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:09:33.599 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:09:35.038 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:09:36.470 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:09:37.859 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:09:39.244 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:09:40.586 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:09:42.050 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:09:44.076 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:09:46.155 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:09:47.980 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:09:49.722 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:09:51.043 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:09:52.715 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:09:54.129 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:09:55.457 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T16:09:55.820 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1926671423,v1:127.0.0.1:6803/1926671423] [v2:127.0.0.1:6804/1926671423,v1:127.0.0.1:6805/1926671423] exists,up c19d82ab-2b6b-4af8-9029-b5038201c89c 2024-09-13T16:09:55.825 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 f61147d5-9ebb-4c78-9bbb-bda5ddb3cee5 2024-09-13T16:09:56.311 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:09:56.372 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:09:56.372+0000 7f80c82f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:09:56.475 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:09:56.476+0000 7f80c82f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:09:56.477 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:09:56.476+0000 7f80c82f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:09:56.477 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:09:56.476+0000 7f80c82f37c0 -1 bdev(0x55983e6c9c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2024-09-13T16:09:56.477 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:09:56.476+0000 7f80c82f37c0 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-13T16:10:04.707 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T16:10:05.099 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T16:10:05.135 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:10:05.136+0000 7f5646ea77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:10:05.175 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:10:05.176+0000 7f5646ea77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:10:05.176 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:10:05.176+0000 7f5646ea77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:10:05.419 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:10:06.490 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:10:06.488+0000 7f5646ea77c0 -1 Falling back to public interface 2024-09-13T16:10:06.720 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:10:08.048 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:10:09.383 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:10:09.508 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:10:09.507+0000 7f5646ea77c0 -1 osd.1 0 log_to_monitors true 2024-09-13T16:10:10.829 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:10:12.314 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:10:13.741 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:10:15.069 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:10:16.483 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:10:18.128 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:10:19.459 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:10:20.859 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:10:22.196 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:10:23.524 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:10:24.864 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:10:26.197 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:10:27.514 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:10:28.866 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:10:30.201 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:10:31.635 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:10:33.227 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:10:34.910 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:10:36.265 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:10:38.207 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:10:39.750 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:10:41.077 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T16:10:42.640 INFO:tasks.workunit.client.0.smithi112.stdout:26 2024-09-13T16:10:43.041 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2502152121,v1:127.0.0.1:6811/2502152121] [v2:127.0.0.1:6812/2502152121,v1:127.0.0.1:6813/2502152121] exists,up f61147d5-9ebb-4c78-9bbb-bda5ddb3cee5 2024-09-13T16:10:43.047 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 5dab3fba-27e1-49ab-a094-225513bb09a8 2024-09-13T16:10:43.463 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:10:43.546 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:10:43.519+0000 7f485f3387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:10:43.590 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:10:43.587+0000 7f485f3387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:10:43.592 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:10:43.591+0000 7f485f3387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:10:43.592 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:10:43.591+0000 7f485f3387c0 -1 bdev(0x55f1ba65bc00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2024-09-13T16:10:43.593 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:10:43.591+0000 7f485f3387c0 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-13T16:10:53.102 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T16:10:53.516 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T16:10:53.552 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:10:53.550+0000 7f9ce0e2c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:10:53.589 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:10:53.586+0000 7f9ce0e2c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:10:53.591 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:10:53.590+0000 7f9ce0e2c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:10:53.822 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:10:54.605 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:10:54.602+0000 7f9ce0e2c7c0 -1 Falling back to public interface 2024-09-13T16:10:55.208 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:10:56.525 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:10:57.823 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:10:57.822+0000 7f9ce0e2c7c0 -1 osd.2 0 log_to_monitors true 2024-09-13T16:10:57.843 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:10:59.164 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:11:00.497 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:11:01.831 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:11:03.326 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:11:04.660 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:11:06.107 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:11:07.436 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:11:08.753 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:11:10.133 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:11:11.535 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:11:12.962 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:11:14.436 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:11:15.880 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:11:17.303 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:11:18.680 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:11:20.187 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:11:21.729 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:11:23.412 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:11:24.906 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:11:26.871 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:11:28.617 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:11:30.456 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T16:11:31.992 INFO:tasks.workunit.client.0.smithi112.stdout:26 2024-09-13T16:11:32.344 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/799027462,v1:127.0.0.1:6819/799027462] [v2:127.0.0.1:6820/799027462,v1:127.0.0.1:6821/799027462] exists,up 5dab3fba-27e1-49ab-a094-225513bb09a8 2024-09-13T16:11:32.788 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'test' created 2024-09-13T16:11:35.212 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836505 2024-09-13T16:11:36.954 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672983 2024-09-13T16:11:37.326 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509447 2024-09-13T16:11:40.010 INFO:tasks.workunit.client.0.smithi112.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-09-13T16:11:40.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: local poolid=1 2024-09-13T16:11:40.011 INFO:tasks.workunit.client.0.smithi112.stdout:Pool: test : 1 2024-09-13T16:11:40.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:478: TEST_just_deep_scrubs: local poolname=test 2024-09-13T16:11:40.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: echo 'Pool: test : 1' 2024-09-13T16:11:40.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:481: TEST_just_deep_scrubs: TESTDATA=testdata.528094 2024-09-13T16:11:40.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: local objects=15 2024-09-13T16:11:40.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:483: TEST_just_deep_scrubs: dd if=/dev/urandom of=testdata.528094 bs=1032 count=1 2024-09-13T16:11:40.013 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T16:11:40.013 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T16:11:40.013 INFO:tasks.workunit.client.0.smithi112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000112695 s, 9.2 MB/s 2024-09-13T16:11:40.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: seq 1 15 2024-09-13T16:11:40.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-13T16:11:40.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rados -p test put obj1 testdata.528094 2024-09-13T16:11:40.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-13T16:11:40.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rados -p test put obj2 testdata.528094 2024-09-13T16:11:40.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-13T16:11:40.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rados -p test put obj3 testdata.528094 2024-09-13T16:11:40.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-13T16:11:40.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rados -p test put obj4 testdata.528094 2024-09-13T16:11:40.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-13T16:11:40.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rados -p test put obj5 testdata.528094 2024-09-13T16:11:41.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-13T16:11:41.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rados -p test put obj6 testdata.528094 2024-09-13T16:11:41.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-13T16:11:41.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rados -p test put obj7 testdata.528094 2024-09-13T16:11:41.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-13T16:11:41.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rados -p test put obj8 testdata.528094 2024-09-13T16:11:41.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-13T16:11:41.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rados -p test put obj9 testdata.528094 2024-09-13T16:11:41.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-13T16:11:41.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rados -p test put obj10 testdata.528094 2024-09-13T16:11:42.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-13T16:11:42.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rados -p test put obj11 testdata.528094 2024-09-13T16:11:42.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-13T16:11:42.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rados -p test put obj12 testdata.528094 2024-09-13T16:11:42.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-13T16:11:42.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rados -p test put obj13 testdata.528094 2024-09-13T16:11:42.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-13T16:11:42.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rados -p test put obj14 testdata.528094 2024-09-13T16:11:43.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-09-13T16:11:43.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rados -p test put obj15 testdata.528094 2024-09-13T16:11:43.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:488: TEST_just_deep_scrubs: rm -f testdata.528094 2024-09-13T16:11:43.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:493: TEST_just_deep_scrubs: ceph osd set noscrub 2024-09-13T16:11:43.713 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T16:11:43.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:494: TEST_just_deep_scrubs: ceph osd set nodeep-scrub 2024-09-13T16:11:44.231 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is set 2024-09-13T16:11:44.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:495: TEST_just_deep_scrubs: sleep 6 2024-09-13T16:11:50.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:496: TEST_just_deep_scrubs: date -Ins 2024-09-13T16:11:50.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:496: TEST_just_deep_scrubs: local now_is=2024-09-13T16:11:50,258859798+00:00 2024-09-13T16:11:50.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:497: TEST_just_deep_scrubs: declare -A sched_data 2024-09-13T16:11:50.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:498: TEST_just_deep_scrubs: local pgid=1.2 2024-09-13T16:11:50.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:501: TEST_just_deep_scrubs: set_query_debug 1.2 2024-09-13T16:11:50.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:303: set_query_debug: local pgid=1.2 2024-09-13T16:11:50.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:305: set_query_debug: ceph pg dump pgs_brief 2024-09-13T16:11:50.259 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:305: set_query_debug: awk -v 'pg=^1.2' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-09-13T16:11:50.566 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs_brief 2024-09-13T16:11:50.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:305: set_query_debug: local prim_osd=0 2024-09-13T16:11:50.584 INFO:tasks.workunit.client.0.smithi112.stdout:Setting scrub debug data. Primary for 1.2 is 0 2024-09-13T16:11:50.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:307: set_query_debug: echo 'Setting scrub debug data. Primary for 1.2 is 0' 2024-09-13T16:11:50.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: set_query_debug: get_asok_path osd.0 2024-09-13T16:11:50.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-13T16:11:50.585 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-13T16:11:50.585 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:11:50.585 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:11:50.586 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:11:50.586 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-osd.0.asok 2024-09-13T16:11:50.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: set_query_debug: CEPH_ARGS= 2024-09-13T16:11:50.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.528094/ceph-osd.0.asok scrubdebug 1.2 set sessions 2024-09-13T16:11:50.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:503: TEST_just_deep_scrubs: extract_published_sch 1.2 2024-09-13T16:11:50,258859798+00:00 2024-09-13T16:11:50,258859798+00:00 sched_data 2024-09-13T16:11:50.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2024-09-13T16:11:50.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2024-09-13T16:11:50.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-09-13T16:11:50,258859798+00:00 2024-09-13T16:11:50.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2024-09-13T16:11:50,258859798+00:00 2024-09-13T16:11:50.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2024-09-13T16:11:50.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-09-13T16:11:50.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-09-13T16:11:50.989 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:11:51.005 INFO:tasks.workunit.client.0.smithi112.stdout:{"success":true}from pg dump pg: { 2024-09-13T16:11:51.005 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_pg_state": "active+clean", 2024-09-13T16:11:51.005 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_state_has_scrubbing": false, 2024-09-13T16:11:51.005 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_last_duration": 0, 2024-09-13T16:11:51.005 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule": "periodic scrub scheduled", 2024-09-13T16:11:51.006 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule_at": "2024-09-14T16:11:32.712695+0000", 2024-09-13T16:11:51.006 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_is_future": true, 2024-09-13T16:11:51.006 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_vs_date": true, 2024-09-13T16:11:51.006 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_reported_epoch": 22, 2024-09-13T16:11:51.006 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_seq": 19 2024-09-13T16:11:51.006 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:11:51.006 INFO:tasks.workunit.client.0.smithi112.stdout:query output: 2024-09-13T16:11:51.142 INFO:tasks.workunit.client.0.smithi112.stdout: "scrubber": { 2024-09-13T16:11:51.142 INFO:tasks.workunit.client.0.smithi112.stdout: "active": false, 2024-09-13T16:11:51.143 INFO:tasks.workunit.client.0.smithi112.stdout: "must_scrub": false, 2024-09-13T16:11:51.143 INFO:tasks.workunit.client.0.smithi112.stdout: "must_deep_scrub": false, 2024-09-13T16:11:51.143 INFO:tasks.workunit.client.0.smithi112.stdout: "must_repair": false, 2024-09-13T16:11:51.143 INFO:tasks.workunit.client.0.smithi112.stdout: "need_auto": false, 2024-09-13T16:11:51.143 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reg_stamp": "2024-09-14T16:11:32.712695+0000", 2024-09-13T16:11:51.143 INFO:tasks.workunit.client.0.smithi112.stdout: "schedule": "scrub scheduled @ 2024-09-14T16:11:32.712 (2024-09-14T16:11:32.712)", 2024-09-13T16:11:51.143 INFO:tasks.workunit.client.0.smithi112.stdout: "test_sequence": 13818880 2024-09-13T16:11:51.143 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:11:51.143 INFO:tasks.workunit.client.0.smithi112.stdout: "agent_state": {} 2024-09-13T16:11:51.306 INFO:tasks.workunit.client.0.smithi112.stdout:( 2024-09-13T16:11:51.306 INFO:tasks.workunit.client.0.smithi112.stdout:[query_epoch]=24 2024-09-13T16:11:51.306 INFO:tasks.workunit.client.0.smithi112.stdout:[query_seq]=23 2024-09-13T16:11:51.306 INFO:tasks.workunit.client.0.smithi112.stdout:[query_active]=false 2024-09-13T16:11:51.306 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule]='scrub scheduled' 2024-09-13T16:11:51.306 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule_at]='2024-09-14T16:11:32.712' 2024-09-13T16:11:51.307 INFO:tasks.workunit.client.0.smithi112.stdout:[query_target_at]='2024-09-14T16:11:32.712' 2024-09-13T16:11:51.307 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_duration]=0 2024-09-13T16:11:51.307 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_stamp]='2024-09-13T16:11:32.712695+0000' 2024-09-13T16:11:51.307 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_scrub]='0x0' 2024-09-13T16:11:51.307 INFO:tasks.workunit.client.0.smithi112.stdout:[query_is_future]=true 2024-09-13T16:11:51.307 INFO:tasks.workunit.client.0.smithi112.stdout:[query_vs_date]=true 2024-09-13T16:11:51.307 INFO:tasks.workunit.client.0.smithi112.stdout:[query_scrub_seq]=13818880 2024-09-13T16:11:51.307 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_pg_state]='active+clean' 2024-09-13T16:11:51.307 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_state_has_scrubbing]=false 2024-09-13T16:11:51.307 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_last_duration]=0 2024-09-13T16:11:51.307 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-09-13T16:11:51.307 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule_at]='2024-09-14T16:11:32.712695+0000' 2024-09-13T16:11:51.307 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_is_future]=true 2024-09-13T16:11:51.308 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_vs_date]=true 2024-09-13T16:11:51.308 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_reported_epoch]=22 2024-09-13T16:11:51.308 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_seq]=19 2024-09-13T16:11:51.308 INFO:tasks.workunit.client.0.smithi112.stdout:) 2024-09-13T16:11:51.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:504: TEST_just_deep_scrubs: local saved_last_stamp=2024-09-13T16:11:32.712695+0000 2024-09-13T16:11:51.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:505: TEST_just_deep_scrubs: local dbg_counter_at_start=13818880 2024-09-13T16:11:51.333 INFO:tasks.workunit.client.0.smithi112.stdout:test counter @ start: 13818880 2024-09-13T16:11:51.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:506: TEST_just_deep_scrubs: echo 'test counter @ start: 13818880' 2024-09-13T16:11:51.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:508: TEST_just_deep_scrubs: ceph tell 1.2 schedule-deep-scrub 2024-09-13T16:11:51.491 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:11:51.491 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T16:11:51.491 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T16:11:51.491 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-08-30T16:10:11.492340+0000" 2024-09-13T16:11:51.491 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:11:51.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:510: TEST_just_deep_scrubs: sleep 5 2024-09-13T16:11:56.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:511: TEST_just_deep_scrubs: declare -A sc_data_2 2024-09-13T16:11:56.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:512: TEST_just_deep_scrubs: extract_published_sch 1.2 2024-09-13T16:11:50,258859798+00:00 2024-09-13T16:11:50,258859798+00:00 sc_data_2 2024-09-13T16:11:56.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2024-09-13T16:11:56.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sc_data_2 2024-09-13T16:11:56.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-09-13T16:11:50,258859798+00:00 2024-09-13T16:11:56.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2024-09-13T16:11:50,258859798+00:00 2024-09-13T16:11:56.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2024-09-13T16:11:56.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-09-13T16:11:56.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-09-13T16:11:56.805 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:11:56.822 INFO:tasks.workunit.client.0.smithi112.stdout:from pg dump pg: { 2024-09-13T16:11:56.822 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_pg_state": "active+clean", 2024-09-13T16:11:56.822 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_state_has_scrubbing": false, 2024-09-13T16:11:56.822 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_last_duration": 0, 2024-09-13T16:11:56.822 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule": "queued for scrub", 2024-09-13T16:11:56.823 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule_at": "0", 2024-09-13T16:11:56.823 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_is_future": false, 2024-09-13T16:11:56.823 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_vs_date": false, 2024-09-13T16:11:56.823 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_reported_epoch": 24, 2024-09-13T16:11:56.823 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_seq": 24 2024-09-13T16:11:56.823 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:11:56.823 INFO:tasks.workunit.client.0.smithi112.stdout:query output: 2024-09-13T16:11:56.969 INFO:tasks.workunit.client.0.smithi112.stdout: "scrubber": { 2024-09-13T16:11:56.969 INFO:tasks.workunit.client.0.smithi112.stdout: "active": false, 2024-09-13T16:11:56.969 INFO:tasks.workunit.client.0.smithi112.stdout: "must_scrub": false, 2024-09-13T16:11:56.969 INFO:tasks.workunit.client.0.smithi112.stdout: "must_deep_scrub": false, 2024-09-13T16:11:56.969 INFO:tasks.workunit.client.0.smithi112.stdout: "must_repair": false, 2024-09-13T16:11:56.969 INFO:tasks.workunit.client.0.smithi112.stdout: "need_auto": false, 2024-09-13T16:11:56.969 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reg_stamp": "2024-09-13T16:11:57.423644+0000", 2024-09-13T16:11:56.969 INFO:tasks.workunit.client.0.smithi112.stdout: "schedule": "deep scrub scheduled @ 2024-09-13T16:11:57.423 (2024-09-06T16:10:11.492)", 2024-09-13T16:11:56.969 INFO:tasks.workunit.client.0.smithi112.stdout: "test_sequence": 13818880 2024-09-13T16:11:56.970 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:11:56.970 INFO:tasks.workunit.client.0.smithi112.stdout: "agent_state": {} 2024-09-13T16:11:57.128 INFO:tasks.workunit.client.0.smithi112.stdout:( 2024-09-13T16:11:57.128 INFO:tasks.workunit.client.0.smithi112.stdout:[query_epoch]=24 2024-09-13T16:11:57.128 INFO:tasks.workunit.client.0.smithi112.stdout:[query_seq]=25 2024-09-13T16:11:57.129 INFO:tasks.workunit.client.0.smithi112.stdout:[query_active]=false 2024-09-13T16:11:57.129 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule]='deep scrub scheduled' 2024-09-13T16:11:57.129 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule_at]='2024-09-13T16:11:57.423' 2024-09-13T16:11:57.129 INFO:tasks.workunit.client.0.smithi112.stdout:[query_target_at]='2024-09-06T16:10:11.492' 2024-09-13T16:11:57.129 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_duration]=0 2024-09-13T16:11:57.129 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_stamp]='2024-08-30T16:10:11.492340+0000' 2024-09-13T16:11:57.129 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_scrub]='0x0' 2024-09-13T16:11:57.129 INFO:tasks.workunit.client.0.smithi112.stdout:[query_is_future]=false 2024-09-13T16:11:57.129 INFO:tasks.workunit.client.0.smithi112.stdout:[query_vs_date]=true 2024-09-13T16:11:57.129 INFO:tasks.workunit.client.0.smithi112.stdout:[query_scrub_seq]=13818880 2024-09-13T16:11:57.129 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_pg_state]='active+clean' 2024-09-13T16:11:57.129 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_state_has_scrubbing]=false 2024-09-13T16:11:57.129 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_last_duration]=0 2024-09-13T16:11:57.129 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule]='queued for scrub' 2024-09-13T16:11:57.130 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule_at]='0' 2024-09-13T16:11:57.130 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_is_future]=false 2024-09-13T16:11:57.130 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_vs_date]=false 2024-09-13T16:11:57.130 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_reported_epoch]=24 2024-09-13T16:11:57.130 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_seq]=24 2024-09-13T16:11:57.130 INFO:tasks.workunit.client.0.smithi112.stdout:) 2024-09-13T16:11:57.158 INFO:tasks.workunit.client.0.smithi112.stdout:test counter @ should show no change: 13818880 2024-09-13T16:11:57.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:513: TEST_just_deep_scrubs: echo 'test counter @ should show no change: ' 13818880 2024-09-13T16:11:57.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:514: TEST_just_deep_scrubs: (( 0 == 0 )) 2024-09-13T16:11:57.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:515: TEST_just_deep_scrubs: (( 13818880 == 13818880 )) 2024-09-13T16:11:57.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:518: TEST_just_deep_scrubs: ceph osd unset nodeep-scrub 2024-09-13T16:11:57.666 INFO:tasks.workunit.client.0.smithi112.stderr:nodeep-scrub is unset 2024-09-13T16:11:57.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:519: TEST_just_deep_scrubs: sleep 5 2024-09-13T16:12:02.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:520: TEST_just_deep_scrubs: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2024-09-13T16:12:02.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:520: TEST_just_deep_scrubs: declare -A expct_qry_duration 2024-09-13T16:12:02.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:521: TEST_just_deep_scrubs: sc_data_2=() 2024-09-13T16:12:02.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:522: TEST_just_deep_scrubs: echo 'test counter @ should be higher than before the unset: ' 2024-09-13T16:12:02.690 INFO:tasks.workunit.client.0.smithi112.stdout:test counter @ should be higher than before the unset: 2024-09-13T16:12:02.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:523: TEST_just_deep_scrubs: wait_any_cond 1.2 10 2024-09-13T16:11:32.712695+0000 expct_qry_duration 'WaitingAfterScrub ' sc_data_2 2024-09-13T16:12:02.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.2 2024-09-13T16:12:02.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2024-09-13T16:12:02.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2024-09-13T16:11:32.712695+0000 2024-09-13T16:12:02.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_qry_duration 2024-09-13T16:12:02.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sc_data_2 2024-09-13T16:12:02.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2024-09-13T16:12:02.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2024-09-13T16:12:02.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2024-09-13T16:12:02.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2024-09-13T16:12:02.692 INFO:tasks.workunit.client.0.smithi112.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.2 dt:2024-09-13T16:11:32.712695+0000 (10 retries) 2024-09-13T16:12:03.506 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:12:03.524 INFO:tasks.workunit.client.0.smithi112.stdout:from pg dump pg: { 2024-09-13T16:12:03.524 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_pg_state": "active+clean", 2024-09-13T16:12:03.524 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_state_has_scrubbing": false, 2024-09-13T16:12:03.524 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_last_duration": 0, 2024-09-13T16:12:03.525 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule": "queued for scrub", 2024-09-13T16:12:03.525 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule_at": "0", 2024-09-13T16:12:03.525 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_is_future": false, 2024-09-13T16:12:03.525 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_vs_date": false, 2024-09-13T16:12:03.525 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_reported_epoch": 25, 2024-09-13T16:12:03.525 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_seq": 26 2024-09-13T16:12:03.525 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:12:03.525 INFO:tasks.workunit.client.0.smithi112.stdout:query output: 2024-09-13T16:12:03.674 INFO:tasks.workunit.client.0.smithi112.stdout: "scrubber": { 2024-09-13T16:12:03.674 INFO:tasks.workunit.client.0.smithi112.stdout: "active": false, 2024-09-13T16:12:03.674 INFO:tasks.workunit.client.0.smithi112.stdout: "must_scrub": false, 2024-09-13T16:12:03.674 INFO:tasks.workunit.client.0.smithi112.stdout: "must_deep_scrub": false, 2024-09-13T16:12:03.675 INFO:tasks.workunit.client.0.smithi112.stdout: "must_repair": false, 2024-09-13T16:12:03.675 INFO:tasks.workunit.client.0.smithi112.stdout: "need_auto": false, 2024-09-13T16:12:03.675 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reg_stamp": "2024-09-13T16:12:02.551970+0000", 2024-09-13T16:12:03.675 INFO:tasks.workunit.client.0.smithi112.stdout: "schedule": "queued for deep scrub at 2024-09-06T16:10:11.492", 2024-09-13T16:12:03.675 INFO:tasks.workunit.client.0.smithi112.stdout: "test_sequence": 13818880 2024-09-13T16:12:03.675 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:03.675 INFO:tasks.workunit.client.0.smithi112.stdout: "agent_state": {} 2024-09-13T16:12:03.851 INFO:tasks.workunit.client.0.smithi112.stdout:( 2024-09-13T16:12:03.851 INFO:tasks.workunit.client.0.smithi112.stdout:[query_epoch]=25 2024-09-13T16:12:03.851 INFO:tasks.workunit.client.0.smithi112.stdout:[query_seq]=27 2024-09-13T16:12:03.851 INFO:tasks.workunit.client.0.smithi112.stdout:[query_active]=false 2024-09-13T16:12:03.851 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule]='queued for deep scrub at 2024-09-06T16:10:11.492' 2024-09-13T16:12:03.852 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule_at]='0' 2024-09-13T16:12:03.852 INFO:tasks.workunit.client.0.smithi112.stdout:[query_target_at]='0' 2024-09-13T16:12:03.852 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_duration]=0 2024-09-13T16:12:03.852 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_stamp]='2024-08-30T16:10:11.492340+0000' 2024-09-13T16:12:03.852 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_scrub]='0x0' 2024-09-13T16:12:03.852 INFO:tasks.workunit.client.0.smithi112.stdout:[query_is_future]=false 2024-09-13T16:12:03.852 INFO:tasks.workunit.client.0.smithi112.stdout:[query_vs_date]=false 2024-09-13T16:12:03.852 INFO:tasks.workunit.client.0.smithi112.stdout:[query_scrub_seq]=13818880 2024-09-13T16:12:03.852 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_pg_state]='active+clean' 2024-09-13T16:12:03.852 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_state_has_scrubbing]=false 2024-09-13T16:12:03.852 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_last_duration]=0 2024-09-13T16:12:03.852 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule]='queued for scrub' 2024-09-13T16:12:03.853 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule_at]='0' 2024-09-13T16:12:03.853 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_is_future]=false 2024-09-13T16:12:03.853 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_vs_date]=false 2024-09-13T16:12:03.853 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_reported_epoch]=25 2024-09-13T16:12:03.853 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_seq]=26 2024-09-13T16:12:03.853 INFO:tasks.workunit.client.0.smithi112.stdout:) 2024-09-13T16:12:03.886 INFO:tasks.workunit.client.0.smithi112.stdout:--> loop: 1 ~ false / 27 / 26 / false / 2024-08-30T16:10:11.492340+0000 / queued for deep scrub at 2024-09-06T16:10:11.492 %%% query_last_duration_neg query_last_duration 2024-09-13T16:12:03.886 INFO:tasks.workunit.client.0.smithi112.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-09-13T16:12:04.697 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:12:04.715 INFO:tasks.workunit.client.0.smithi112.stdout:from pg dump pg: { 2024-09-13T16:12:04.715 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_pg_state": "active+clean", 2024-09-13T16:12:04.715 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_state_has_scrubbing": false, 2024-09-13T16:12:04.715 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_last_duration": 0, 2024-09-13T16:12:04.715 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule": "queued for scrub", 2024-09-13T16:12:04.715 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_schedule_at": "0", 2024-09-13T16:12:04.715 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_is_future": false, 2024-09-13T16:12:04.715 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_vs_date": false, 2024-09-13T16:12:04.715 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_reported_epoch": 25, 2024-09-13T16:12:04.715 INFO:tasks.workunit.client.0.smithi112.stdout: "dmp_seq": 26 2024-09-13T16:12:04.715 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:12:04.716 INFO:tasks.workunit.client.0.smithi112.stdout:query output: 2024-09-13T16:12:04.900 INFO:tasks.workunit.client.0.smithi112.stdout: "scrubber": { 2024-09-13T16:12:04.901 INFO:tasks.workunit.client.0.smithi112.stdout: "active": true, 2024-09-13T16:12:04.901 INFO:tasks.workunit.client.0.smithi112.stdout: "epoch_start": "17", 2024-09-13T16:12:04.901 INFO:tasks.workunit.client.0.smithi112.stdout: "start": "MIN", 2024-09-13T16:12:04.901 INFO:tasks.workunit.client.0.smithi112.stdout: "end": "MIN", 2024-09-13T16:12:04.901 INFO:tasks.workunit.client.0.smithi112.stdout: "max_end": "MIN", 2024-09-13T16:12:04.901 INFO:tasks.workunit.client.0.smithi112.stdout: "subset_last_update": "0'0", 2024-09-13T16:12:04.901 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": true, 2024-09-13T16:12:04.901 INFO:tasks.workunit.client.0.smithi112.stdout: "req_scrub": false, 2024-09-13T16:12:04.901 INFO:tasks.workunit.client.0.smithi112.stdout: "auto_repair": false, 2024-09-13T16:12:04.901 INFO:tasks.workunit.client.0.smithi112.stdout: "check_repair": false, 2024-09-13T16:12:04.901 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_scrub_on_error": false, 2024-09-13T16:12:04.902 INFO:tasks.workunit.client.0.smithi112.stdout: "priority": 0, 2024-09-13T16:12:04.902 INFO:tasks.workunit.client.0.smithi112.stdout: "shallow_errors": 0, 2024-09-13T16:12:04.902 INFO:tasks.workunit.client.0.smithi112.stdout: "deep_errors": 0, 2024-09-13T16:12:04.902 INFO:tasks.workunit.client.0.smithi112.stdout: "fixed": 0, 2024-09-13T16:12:04.902 INFO:tasks.workunit.client.0.smithi112.stdout: "waiting_on_whom": [], 2024-09-13T16:12:04.902 INFO:tasks.workunit.client.0.smithi112.stdout: "schedule": "scrubbing", 2024-09-13T16:12:04.902 INFO:tasks.workunit.client.0.smithi112.stdout: "test_sequence": 13818883 2024-09-13T16:12:04.902 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:04.902 INFO:tasks.workunit.client.0.smithi112.stdout: "agent_state": {} 2024-09-13T16:12:05.068 INFO:tasks.workunit.client.0.smithi112.stdout:( 2024-09-13T16:12:05.069 INFO:tasks.workunit.client.0.smithi112.stdout:[query_epoch]=25 2024-09-13T16:12:05.069 INFO:tasks.workunit.client.0.smithi112.stdout:[query_seq]=35 2024-09-13T16:12:05.069 INFO:tasks.workunit.client.0.smithi112.stdout:[query_active]=true 2024-09-13T16:12:05.069 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule]='scrubbing' 2024-09-13T16:12:05.069 INFO:tasks.workunit.client.0.smithi112.stdout:[query_schedule_at]='0' 2024-09-13T16:12:05.069 INFO:tasks.workunit.client.0.smithi112.stdout:[query_target_at]='0' 2024-09-13T16:12:05.069 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_duration]=1 2024-09-13T16:12:05.069 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_stamp]='2024-09-13T16:12:04.557961+0000' 2024-09-13T16:12:05.069 INFO:tasks.workunit.client.0.smithi112.stdout:[query_last_scrub]='0x0' 2024-09-13T16:12:05.069 INFO:tasks.workunit.client.0.smithi112.stdout:[query_is_future]=false 2024-09-13T16:12:05.069 INFO:tasks.workunit.client.0.smithi112.stdout:[query_vs_date]=false 2024-09-13T16:12:05.069 INFO:tasks.workunit.client.0.smithi112.stdout:[query_scrub_seq]=13818883 2024-09-13T16:12:05.070 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_pg_state]='active+clean' 2024-09-13T16:12:05.070 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_state_has_scrubbing]=false 2024-09-13T16:12:05.070 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_last_duration]=0 2024-09-13T16:12:05.070 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule]='queued for scrub' 2024-09-13T16:12:05.070 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_schedule_at]='0' 2024-09-13T16:12:05.070 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_is_future]=false 2024-09-13T16:12:05.070 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_vs_date]=false 2024-09-13T16:12:05.070 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_reported_epoch]=25 2024-09-13T16:12:05.070 INFO:tasks.workunit.client.0.smithi112.stdout:[dmp_seq]=26 2024-09-13T16:12:05.070 INFO:tasks.workunit.client.0.smithi112.stdout:) 2024-09-13T16:12:05.103 INFO:tasks.workunit.client.0.smithi112.stdout:--> loop: 2 ~ true / 35 / 26 / false / 2024-09-13T16:12:04.557961+0000 / scrubbing %%% query_last_duration_neg query_last_duration 2024-09-13T16:12:05.103 INFO:tasks.workunit.client.0.smithi112.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 1 2024-09-13T16:12:05.104 INFO:tasks.workunit.client.0.smithi112.stdout:WaitingAfterScrub - 'query_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-09-13T16:12:05.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2024-09-13T16:12:05.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:524: TEST_just_deep_scrubs: perf_counters td/osd-scrub-test 3 2024-09-13T16:12:05.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-09-13T16:12:05.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2024-09-13T16:12:05.105 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2024-09-13T16:12:05.106 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2024-09-13T16:12:05.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:12:05.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-09-13T16:12:05.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:12:05.234 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:12:05.234 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:12:05.234 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:12:05.234 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:12:05.234 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:12:05.234 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:12:05.234 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.234 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:12:05.234 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:12:05.234 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:12:05.234 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:12:05.235 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:12:05.235 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.235 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.235 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.235 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.235 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:12:05.235 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:12:05.235 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.235 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.235 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.235 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.235 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:12:05.235 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:12:05.236 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:12:05.236 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:12:05.236 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:12:05.236 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:12:05.236 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:12:05.236 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.236 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.236 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.236 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.236 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:12:05.236 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:12:05.236 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:12:05.237 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:12:05.237 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.237 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.237 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.237 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.237 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:12:05.237 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.237 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.237 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:12:05.237 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:12:05.237 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:12:05.237 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:12:05.237 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:12:05.238 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:12:05.238 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.238 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:12:05.238 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 1, 2024-09-13T16:12:05.238 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 1, 2024-09-13T16:12:05.238 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 1, 2024-09-13T16:12:05.238 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:12:05.238 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 1, 2024-09-13T16:12:05.238 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0.003999897, 2024-09-13T16:12:05.238 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0.003999897 2024-09-13T16:12:05.238 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.238 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:12:05.238 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:12:05.239 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.239 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.239 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.239 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.239 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:12:05.239 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 1, 2024-09-13T16:12:05.239 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:12:05.239 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:12:05.239 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:12:05.239 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 1, 2024-09-13T16:12:05.239 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:12:05.239 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 1, 2024-09-13T16:12:05.239 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.239 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.240 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.240 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:12:05.240 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:12:05.240 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:12:05.240 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:12:05.240 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.240 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.240 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.240 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.240 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 2 2024-09-13T16:12:05.240 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.240 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.240 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:12:05.241 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:12:05.241 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:12:05.241 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:12:05.241 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:12:05.241 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:12:05.241 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.241 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:12:05.241 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:12:05.241 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:12:05.241 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:12:05.241 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:12:05.241 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.241 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.242 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.242 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.242 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:12:05.242 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:12:05.242 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.242 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.242 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.242 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.242 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:12:05.242 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:12:05.242 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:12:05.242 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:12:05.242 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:12:05.243 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:12:05.243 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:12:05.243 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.243 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.243 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.243 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.243 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:12:05.243 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:12:05.243 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:12:05.243 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:12:05.243 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.243 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.243 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.243 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.243 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:12:05.244 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.244 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.244 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:12:05.244 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:12:05.244 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:12:05.244 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:12:05.244 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:12:05.244 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:12:05.244 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.244 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:12:05.244 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:12:05.244 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:12:05.244 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:12:05.245 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:12:05.245 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.245 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.245 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.245 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.245 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:12:05.245 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:12:05.245 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.245 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.245 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.245 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.245 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:12:05.245 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:12:05.245 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:12:05.246 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:12:05.246 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:12:05.246 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:12:05.246 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:12:05.246 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.246 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.246 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.246 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.247 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:12:05.247 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:12:05.247 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:12:05.247 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:12:05.247 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.247 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.247 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.247 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.247 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:12:05.247 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.247 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.247 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:12:05.247 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:12:05.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:12:05.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-09-13T16:12:05.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:12:05.377 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:12:05.378 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:12:05.378 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:12:05.378 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:12:05.378 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:12:05.378 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:12:05.378 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.378 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:12:05.378 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:12:05.378 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:12:05.378 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:12:05.378 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:12:05.378 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.378 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.379 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.379 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.379 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:12:05.379 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:12:05.379 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.379 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.379 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.379 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.379 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:12:05.379 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:12:05.379 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:12:05.379 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:12:05.379 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:12:05.379 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:12:05.380 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:12:05.380 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.380 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.380 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.380 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.380 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:12:05.380 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:12:05.380 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:12:05.380 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:12:05.380 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.380 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.380 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.380 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.381 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:12:05.381 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.381 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.381 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:12:05.381 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:12:05.381 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:12:05.381 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:12:05.381 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:12:05.381 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:12:05.381 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.381 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:12:05.381 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:12:05.381 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:12:05.382 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:12:05.382 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:12:05.382 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.382 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.382 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.382 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.382 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:12:05.382 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:12:05.382 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.382 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.382 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.382 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.382 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:12:05.382 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:12:05.383 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:12:05.383 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:12:05.383 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:12:05.383 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:12:05.383 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:12:05.383 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.383 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.383 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.383 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.383 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:12:05.383 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:12:05.383 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:12:05.383 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:12:05.384 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.384 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.384 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.384 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.384 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:12:05.384 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.384 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.384 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:12:05.384 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:12:05.384 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:12:05.384 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:12:05.384 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:12:05.384 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:12:05.384 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.385 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:12:05.385 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:12:05.385 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:12:05.385 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:12:05.385 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:12:05.385 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.385 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.385 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.385 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.385 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:12:05.385 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:12:05.385 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.386 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.386 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.386 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.386 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:12:05.386 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:12:05.386 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:12:05.386 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:12:05.386 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:12:05.386 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:12:05.386 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:12:05.386 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.386 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.386 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.387 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.387 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:12:05.387 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:12:05.387 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:12:05.387 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:12:05.387 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.387 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.387 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.387 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.387 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:12:05.387 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.387 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.387 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:12:05.387 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:12:05.388 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:12:05.388 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:12:05.388 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:12:05.388 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:12:05.388 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.388 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:12:05.388 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:12:05.388 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:12:05.388 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:12:05.388 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:12:05.388 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.388 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.388 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.388 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.389 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:12:05.389 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:12:05.389 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.389 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.389 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.389 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.389 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:12:05.389 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:12:05.389 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:12:05.389 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:12:05.389 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:12:05.389 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:12:05.389 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:12:05.390 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.390 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.390 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.390 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.390 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:12:05.390 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:12:05.390 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:12:05.390 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:12:05.390 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.391 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.391 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.391 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.391 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:12:05.391 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.391 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.391 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:12:05.391 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:12:05.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:12:05.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2024-09-13T16:12:05.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:12:05.518 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:12:05.518 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:12:05.518 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:12:05.518 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:12:05.518 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:12:05.518 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:12:05.519 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.519 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:12:05.519 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:12:05.519 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:12:05.519 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:12:05.519 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:12:05.519 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.519 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.519 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.519 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.519 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:12:05.519 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:12:05.520 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.520 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.520 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.520 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.520 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:12:05.520 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:12:05.520 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:12:05.520 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:12:05.520 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:12:05.520 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:12:05.520 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:12:05.520 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.520 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.521 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.521 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.521 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:12:05.521 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:12:05.521 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:12:05.521 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:12:05.521 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.521 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.521 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.521 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.521 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:12:05.521 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.522 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.522 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:12:05.522 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:12:05.522 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:12:05.522 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:12:05.522 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:12:05.522 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:12:05.522 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.522 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:12:05.522 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:12:05.522 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:12:05.522 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:12:05.522 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:12:05.523 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.523 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.523 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.523 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.523 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:12:05.523 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:12:05.523 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.523 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.523 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.523 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.523 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:12:05.523 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:12:05.523 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:12:05.523 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:12:05.524 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:12:05.524 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:12:05.524 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:12:05.524 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.524 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.524 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.524 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.524 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:12:05.524 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:12:05.524 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:12:05.525 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:12:05.525 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.525 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.525 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.525 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.525 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:12:05.525 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.525 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.525 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:12:05.525 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:12:05.525 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:12:05.525 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:12:05.526 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:12:05.526 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:12:05.526 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.526 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:12:05.526 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:12:05.526 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:12:05.526 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:12:05.526 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:12:05.526 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.526 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.527 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.527 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.527 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:12:05.527 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:12:05.527 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.527 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.527 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.527 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.527 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:12:05.527 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:12:05.527 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:12:05.527 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:12:05.527 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:12:05.527 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:12:05.528 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:12:05.528 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.528 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.528 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.528 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.528 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:12:05.528 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:12:05.528 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:12:05.528 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:12:05.528 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.528 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.528 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.528 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.528 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:12:05.529 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.529 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.529 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:12:05.529 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:12:05.529 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:12:05.529 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:12:05.529 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:12:05.529 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:12:05.529 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.529 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:12:05.529 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:12:05.529 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:12:05.529 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:12:05.530 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:12:05.530 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.530 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.530 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.530 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.530 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:12:05.530 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:12:05.530 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.530 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.530 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.530 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.530 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:12:05.530 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:12:05.530 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:12:05.531 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:12:05.531 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:12:05.531 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:12:05.531 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:12:05.531 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.531 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.532 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.532 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.532 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:12:05.532 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:12:05.532 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:12:05.532 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:12:05.532 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:12:05.532 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:12:05.532 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:12:05.532 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:12:05.532 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:12:05.532 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.532 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:12:05.532 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:12:05.533 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:12:05.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_just_deep_scrubs ------------------' 2024-09-13T16:12:05.533 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Teardown Test TEST_just_deep_scrubs ------------------ 2024-09-13T16:12:05.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-09-13T16:12:05.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:12:05.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:12:05.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:12:05.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:12:05.534 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:12:05.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:12:05.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:12:05.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:12:05.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:12:05.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:12:05.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:12:05.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:12:05.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:12:05.845 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:12:05.846 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:12:05.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:12:05.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:12:05.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:12:05.847 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:12:05.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:12:05.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:12:05.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:12:05.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:12:05.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:12:05.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:12:05.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:12:05.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:12:05.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:12:05.872 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Complete Test TEST_just_deep_scrubs ------------------ 2024-09-13T16:12:05.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_just_deep_scrubs ------------------' 2024-09-13T16:12:05.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-13T16:12:05.873 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Prepare Test TEST_pg_dump_objects_scrubbed ------------------- 2024-09-13T16:12:05.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_pg_dump_objects_scrubbed -------------------' 2024-09-13T16:12:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-09-13T16:12:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2024-09-13T16:12:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2024-09-13T16:12:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:12:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:12:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:12:05.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:12:05.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:12:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:12:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:12:05.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:12:05.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:12:05.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:12:05.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:12:05.878 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:12:05.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:12:05.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:12:05.879 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:12:05.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:12:05.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:12:05.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:12:05.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:12:05.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:12:05.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:12:05.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:12:05.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:12:05.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:12:05.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:12:05.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:12:05.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:12:05.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:12:05.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2024-09-13T16:12:05.888 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T16:12:05.888 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:12:05.888 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:12:05.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.528094 2024-09-13T16:12:05.889 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T16:12:05.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T16:12:05.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T16:12:05.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-13T16:12:05.890 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Run Test TEST_pg_dump_objects_scrubbed ----------------------- 2024-09-13T16:12:05.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_pg_dump_objects_scrubbed -----------------------' 2024-09-13T16:12:05.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_pg_dump_objects_scrubbed td/osd-scrub-test 2024-09-13T16:12:05.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:648: TEST_pg_dump_objects_scrubbed: local dir=td/osd-scrub-test 2024-09-13T16:12:05.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:649: TEST_pg_dump_objects_scrubbed: local poolname=test 2024-09-13T16:12:05.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:650: TEST_pg_dump_objects_scrubbed: local OSDS=3 2024-09-13T16:12:05.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:651: TEST_pg_dump_objects_scrubbed: local objects=15 2024-09-13T16:12:05.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:652: TEST_pg_dump_objects_scrubbed: local timeout=10 2024-09-13T16:12:05.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:654: TEST_pg_dump_objects_scrubbed: TESTDATA=testdata.528094 2024-09-13T16:12:05.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:656: TEST_pg_dump_objects_scrubbed: setup td/osd-scrub-test 2024-09-13T16:12:05.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2024-09-13T16:12:05.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2024-09-13T16:12:05.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:12:05.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:12:05.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:12:05.893 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:12:05.893 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:12:05.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:12:05.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:12:05.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:12:05.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:12:05.895 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:12:05.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:12:05.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:12:05.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:12:05.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:12:05.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:12:05.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:12:05.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:12:05.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:12:05.899 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:12:05.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:12:05.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:12:05.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:12:05.902 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:12:05.903 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:12:05.903 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:12:05.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:12:05.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:12:05.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:12:05.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2024-09-13T16:12:05.905 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T16:12:05.906 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:12:05.906 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:12:05.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.528094 2024-09-13T16:12:05.908 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T16:12:05.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T16:12:05.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T16:12:05.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-13T16:12:05.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:657: TEST_pg_dump_objects_scrubbed: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-13T16:12:05.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2024-09-13T16:12:05.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T16:12:05.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T16:12:05.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T16:12:05.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2024-09-13T16:12:05.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-13T16:12:06.442 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T16:12:06.442 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:12:06.442 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:12:06.442 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:12:06.443 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:12:06.443 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:12:06.443 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:12:06.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-13T16:12:06.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T16:12:06.697 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T16:12:06.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T16:12:06.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T16:12:06.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T16:12:06.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T16:12:06.698 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T16:12:06.699 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T16:12:06.699 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T16:12:06.699 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:12:06.699 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:12:06.699 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:12:06.699 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-mon.a.asok 2024-09-13T16:12:06.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T16:12:06.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.528094/ceph-mon.a.asok config get fsid 2024-09-13T16:12:06.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T16:12:06.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T16:12:06.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T16:12:06.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T16:12:06.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T16:12:06.816 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T16:12:06.816 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T16:12:06.816 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T16:12:06.817 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:12:06.817 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:12:06.817 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:12:06.817 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-mon.a.asok 2024-09-13T16:12:06.817 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T16:12:06.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.528094/ceph-mon.a.asok config get mon_host 2024-09-13T16:12:06.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:658: TEST_pg_dump_objects_scrubbed: run_mgr td/osd-scrub-test x 2024-09-13T16:12:06.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2024-09-13T16:12:06.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T16:12:06.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T16:12:06.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T16:12:06.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2024-09-13T16:12:06.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T16:12:07.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T16:12:07.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:12:07.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:12:07.871 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:12:07.871 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:12:07.871 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:12:07.872 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:12:07.872 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T16:12:07.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T16:12:07.921 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:659: TEST_pg_dump_objects_scrubbed: expr 3 - 1 2024-09-13T16:12:07.921 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:659: TEST_pg_dump_objects_scrubbed: seq 0 2 2024-09-13T16:12:07.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:659: TEST_pg_dump_objects_scrubbed: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:12:07.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:661: TEST_pg_dump_objects_scrubbed: run_osd td/osd-scrub-test 0 2024-09-13T16:12:07.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:12:07.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:12:07.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T16:12:07.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:12:07.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-13T16:12:07.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:12:07.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:12:07.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:12:07.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:12:07.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-13T16:12:07.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-13T16:12:07.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:12:07.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:12:07.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:12:07.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:12:07.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:12:07.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:12:07.927 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:12:07.927 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:12:07.927 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:12:07.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:12:07.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:12:07.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:12:07.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:12:07.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:12:07.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:12:07.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:12:07.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:12:07.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:12:07.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:12:07.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:12:07.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:12:07.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T16:12:07.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-13T16:12:07.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:12:07.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0a6371e2-e1b3-49be-98e6-4ab16c92f0b5 2024-09-13T16:12:07.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0a6371e2-e1b3-49be-98e6-4ab16c92f0b5' 2024-09-13T16:12:07.930 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 0a6371e2-e1b3-49be-98e6-4ab16c92f0b5 2024-09-13T16:12:07.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:12:07.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBXZORmhHVTOBAArpxpEIK5r/RqBTlBdg2j6A== 2024-09-13T16:12:07.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBXZORmhHVTOBAArpxpEIK5r/RqBTlBdg2j6A=="}' 2024-09-13T16:12:07.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0a6371e2-e1b3-49be-98e6-4ab16c92f0b5 -i td/osd-scrub-test/0/new.json 2024-09-13T16:12:08.223 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:12:08.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-13T16:12:08.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXZORmhHVTOBAArpxpEIK5r/RqBTlBdg2j6A== --osd-uuid 0a6371e2-e1b3-49be-98e6-4ab16c92f0b5 2024-09-13T16:12:08.277 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:12:08.272+0000 7f288d0257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:12:08.316 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:12:08.312+0000 7f288d0257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:12:08.318 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:12:08.312+0000 7f288d0257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:12:08.318 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:12:08.312+0000 7f288d0257c0 -1 bdev(0x5587c1653c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2024-09-13T16:12:08.318 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:12:08.312+0000 7f288d0257c0 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-13T16:12:16.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-13T16:12:16.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:12:16.404 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T16:12:16.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T16:12:16.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:12:16.817 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T16:12:16.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T16:12:16.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:12:16.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T16:12:16.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:12:16.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:12:16.847 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:12:16.844+0000 7fb76d51e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:12:16.853 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:12:16.844+0000 7fb76d51e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:12:16.853 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:12:16.848+0000 7fb76d51e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:12:17.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T16:12:17.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:12:17.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T16:12:17.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:12:17.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:12:17.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:17.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:12:17.186 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:12:17.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:17.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:17.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:18.117 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:12:18.116+0000 7fb76d51e7c0 -1 Falling back to public interface 2024-09-13T16:12:18.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:18.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:18.501 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:12:18.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:12:18.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:18.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:18.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:19.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:19.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:19.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:12:19.827 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:12:19.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:19.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:20.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:21.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:21.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:21.149 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:12:21.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:12:21.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:21.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:21.262 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:12:21.260+0000 7fb76d51e7c0 -1 osd.0 0 log_to_monitors true 2024-09-13T16:12:21.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:22.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:22.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:22.524 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:12:22.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:12:22.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:22.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:22.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:23.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:23.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:23.853 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:12:23.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:12:23.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:23.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:24.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:25.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:25.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:25.366 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:12:25.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T16:12:25.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:25.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:25.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:26.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:26.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:26.684 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:12:26.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T16:12:26.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:26.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:27.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:28.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:28.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:28.065 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:12:28.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T16:12:28.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:28.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:28.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:29.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:29.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:29.524 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:12:29.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T16:12:29.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:29.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:29.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:30.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:30.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:30.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T16:12:30.844 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:12:30.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:30.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:31.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:32.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:32.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:32.365 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:12:32.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T16:12:32.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:32.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:32.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:33.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:33.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:33.705 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:12:33.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T16:12:33.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:33.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:34.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:35.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:35.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:35.097 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:12:35.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T16:12:35.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:35.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:35.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:36.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:36.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:36.558 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:12:36.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T16:12:36.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:36.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:37.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:38.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:38.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:38.062 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:12:38.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T16:12:38.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:38.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:38.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:39.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:39.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:39.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T16:12:39.626 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:12:39.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:39.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:39.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:40.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:40.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:40.940 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:12:40.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T16:12:40.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:40.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:41.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:42.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T16:12:42.493 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:12:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:42.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:42.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:43.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:43.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:43.937 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:12:43.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T16:12:43.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:43.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:44.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:45.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:45.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:45.427 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:12:45.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T16:12:45.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:45.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:46.198 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:47.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:47.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:47.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T16:12:47.201 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:12:47.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:47.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:48.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:49.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:49.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:49.273 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:12:49.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T16:12:49.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:49.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:49.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:51.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:51.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:51.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T16:12:51.002 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:12:51.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:51.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:51.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:12:52.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:12:52.991 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:12:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:12:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T16:12:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:12:52.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:12:53.440 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4198460990,v1:127.0.0.1:6803/4198460990] [v2:127.0.0.1:6804/4198460990,v1:127.0.0.1:6805/4198460990] exists,up 0a6371e2-e1b3-49be-98e6-4ab16c92f0b5 2024-09-13T16:12:53.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:12:53.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:12:53.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:12:53.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:659: TEST_pg_dump_objects_scrubbed: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:12:53.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:661: TEST_pg_dump_objects_scrubbed: run_osd td/osd-scrub-test 1 2024-09-13T16:12:53.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:12:53.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:12:53.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T16:12:53.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:12:53.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-13T16:12:53.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:12:53.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:12:53.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:12:53.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:12:53.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-13T16:12:53.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-13T16:12:53.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:12:53.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:12:53.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:12:53.445 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:12:53.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:12:53.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:12:53.446 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:12:53.446 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:12:53.446 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:12:53.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:12:53.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:12:53.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:12:53.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:12:53.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:12:53.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:12:53.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:12:53.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:12:53.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:12:53.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:12:53.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:12:53.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:12:53.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T16:12:53.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-13T16:12:53.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:12:53.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=76972f7a-b612-4f98-97b7-3735df85b026 2024-09-13T16:12:53.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 76972f7a-b612-4f98-97b7-3735df85b026' 2024-09-13T16:12:53.448 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 76972f7a-b612-4f98-97b7-3735df85b026 2024-09-13T16:12:53.448 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:12:53.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCFZORmItrWGxAADqVtOVvF2493kotH4fHoSw== 2024-09-13T16:12:53.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCFZORmItrWGxAADqVtOVvF2493kotH4fHoSw=="}' 2024-09-13T16:12:53.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 76972f7a-b612-4f98-97b7-3735df85b026 -i td/osd-scrub-test/1/new.json 2024-09-13T16:12:53.921 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:12:53.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-13T16:12:53.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFZORmItrWGxAADqVtOVvF2493kotH4fHoSw== --osd-uuid 76972f7a-b612-4f98-97b7-3735df85b026 2024-09-13T16:12:53.974 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:12:53.971+0000 7f80005e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:12:54.064 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:12:54.063+0000 7f80005e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:12:54.066 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:12:54.063+0000 7f80005e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:12:54.066 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:12:54.063+0000 7f80005e87c0 -1 bdev(0x557a78891c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2024-09-13T16:12:54.066 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:12:54.067+0000 7f80005e87c0 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-13T16:13:02.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-13T16:13:02.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:13:02.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T16:13:02.680 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T16:13:02.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:13:03.079 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T16:13:03.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T16:13:03.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:13:03.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T16:13:03.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:13:03.081 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:13:03.110 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:13:03.107+0000 7f3762d857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:13:03.121 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:13:03.119+0000 7f3762d857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:13:03.123 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:13:03.123+0000 7f3762d857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:13:03.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T16:13:03.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:13:03.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T16:13:03.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:13:03.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:13:03.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:03.461 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:13:03.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:13:03.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:03.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:03.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:04.233 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:13:04.231+0000 7f3762d857c0 -1 Falling back to public interface 2024-09-13T16:13:04.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:04.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:04.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:13:04.814 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:13:04.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:04.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:05.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:06.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:06.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:06.130 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:13:06.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:13:06.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:06.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:06.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:07.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:07.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:07.462 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:13:07.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:13:07.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:07.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:07.502 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:13:07.499+0000 7f3762d857c0 -1 osd.1 0 log_to_monitors true 2024-09-13T16:13:07.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:08.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:08.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:08.816 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:13:08.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:13:08.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:08.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:09.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:10.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:10.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:10.181 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:13:10.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:13:10.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:10.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:10.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:11.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:11.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:11.618 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:13:11.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T16:13:11.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:11.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:12.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:13.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:13.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:13.140 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:13:13.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T16:13:13.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:13.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:13.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:14.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:14.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:14.552 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:13:14.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T16:13:14.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:14.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:14.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:15.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:15.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:15.946 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:13:15.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T16:13:15.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:15.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:16.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:17.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:17.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:17.348 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:13:17.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T16:13:17.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:17.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:17.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:18.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:18.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:18.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T16:13:18.672 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:13:18.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:18.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:19.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:20.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:20.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:20.143 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:13:20.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T16:13:20.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:20.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:20.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:21.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:21.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:21.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T16:13:21.644 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:13:21.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:21.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:22.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:23.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:23.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:23.101 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:13:23.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T16:13:23.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:23.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:23.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:24.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:24.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:24.521 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:13:24.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T16:13:24.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:24.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:24.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:25.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:25.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:25.856 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:13:25.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T16:13:25.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:25.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:26.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:27.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:27.190 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:27.190 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:13:27.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T16:13:27.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:27.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:27.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:28.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:28.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:28.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T16:13:28.876 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:13:28.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:28.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:29.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:30.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:30.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:30.541 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:13:30.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T16:13:30.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:30.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:31.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:32.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:32.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:32.269 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:13:32.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T16:13:32.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:32.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:33.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:34.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:34.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:34.305 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:13:34.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T16:13:34.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:34.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:35.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:36.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:36.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:36.140 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:13:36.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T16:13:36.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:36.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:36.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:37.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:37.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:37.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T16:13:37.748 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:13:37.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:37.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:38.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:39.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:39.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:39.303 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:13:39.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T16:13:39.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:39.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:13:39.756 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/493670091,v1:127.0.0.1:6811/493670091] [v2:127.0.0.1:6812/493670091,v1:127.0.0.1:6813/493670091] exists,up 76972f7a-b612-4f98-97b7-3735df85b026 2024-09-13T16:13:39.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:13:39.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:13:39.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:13:39.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:659: TEST_pg_dump_objects_scrubbed: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:13:39.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:661: TEST_pg_dump_objects_scrubbed: run_osd td/osd-scrub-test 2 2024-09-13T16:13:39.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:13:39.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:13:39.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T16:13:39.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:13:39.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-13T16:13:39.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:13:39.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:13:39.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:13:39.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:13:39.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-13T16:13:39.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-13T16:13:39.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:13:39.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:13:39.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:13:39.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:13:39.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:13:39.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:13:39.760 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:13:39.760 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:13:39.760 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:13:39.762 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:13:39.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:13:39.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:13:39.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:13:39.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:13:39.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:13:39.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:13:39.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:13:39.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:13:39.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:13:39.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:13:39.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:13:39.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T16:13:39.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-13T16:13:39.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:13:39.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1e92deb4-fd93-4ca3-b1a0-c3adba3258c0 2024-09-13T16:13:39.763 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 1e92deb4-fd93-4ca3-b1a0-c3adba3258c0 2024-09-13T16:13:39.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 1e92deb4-fd93-4ca3-b1a0-c3adba3258c0' 2024-09-13T16:13:39.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:13:39.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCzZORmTjimLhAA6MJrxldSmdmWtutQHYXbQQ== 2024-09-13T16:13:39.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCzZORmTjimLhAA6MJrxldSmdmWtutQHYXbQQ=="}' 2024-09-13T16:13:39.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1e92deb4-fd93-4ca3-b1a0-c3adba3258c0 -i td/osd-scrub-test/2/new.json 2024-09-13T16:13:40.274 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:13:40.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-13T16:13:40.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCzZORmTjimLhAA6MJrxldSmdmWtutQHYXbQQ== --osd-uuid 1e92deb4-fd93-4ca3-b1a0-c3adba3258c0 2024-09-13T16:13:40.330 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:13:40.326+0000 7fb862b967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:13:40.410 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:13:40.406+0000 7fb862b967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:13:40.412 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:13:40.410+0000 7fb862b967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:13:40.413 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:13:40.410+0000 7fb862b967c0 -1 bdev(0x561b15b91c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2024-09-13T16:13:40.413 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:13:40.410+0000 7fb862b967c0 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-13T16:13:46.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-13T16:13:46.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:13:46.884 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T16:13:46.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T16:13:46.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:13:47.329 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T16:13:47.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T16:13:47.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T16:13:47.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:13:47.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:13:47.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:13:47.359 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:13:47.358+0000 7ff5fb1e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:13:47.398 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:13:47.394+0000 7ff5fb1e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:13:47.399 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:13:47.398+0000 7ff5fb1e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:13:47.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T16:13:47.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:13:47.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T16:13:47.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:13:47.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:13:47.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:47.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:13:47.655 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:13:47.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:47.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:13:47.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:48.925 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:13:48.922+0000 7ff5fb1e77c0 -1 Falling back to public interface 2024-09-13T16:13:48.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:48.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:48.968 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:13:48.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:13:48.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:48.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:13:49.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:50.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:50.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:50.340 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:13:50.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:13:50.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:50.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:13:50.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:51.574 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:13:51.570+0000 7ff5fb1e77c0 -1 osd.2 0 log_to_monitors true 2024-09-13T16:13:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:51.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:13:51.713 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:13:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:13:52.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:53.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:53.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:53.223 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:13:53.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:13:53.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:53.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:13:53.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:54.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:54.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:54.646 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:13:54.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:13:54.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:54.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:13:54.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:55.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:55.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:55.971 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:13:55.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T16:13:55.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:55.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:13:56.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:57.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:57.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:57.363 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:13:57.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T16:13:57.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:57.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:13:57.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:13:58.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:13:58.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:13:58.742 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:13:58.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T16:13:58.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:13:58.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:13:59.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:00.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:00.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:00.074 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:14:00.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T16:14:00.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:00.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:14:00.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:01.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:01.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:01.396 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:14:01.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T16:14:01.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:01.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:14:01.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:02.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:02.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:02.727 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:14:02.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T16:14:02.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:02.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:14:03.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:04.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:04.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:04.076 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:14:04.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T16:14:04.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:04.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:14:04.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:05.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:05.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:05.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T16:14:05.414 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:14:05.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:05.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:14:05.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:06.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:06.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:06.921 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:14:06.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T16:14:06.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:06.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:14:07.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:08.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:08.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:08.820 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:14:08.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T16:14:08.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:08.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:14:09.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:10.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:10.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:10.285 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:14:10.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T16:14:10.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:10.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:14:10.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:11.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:11.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:11.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T16:14:11.744 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:14:11.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:11.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:14:12.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:13.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:13.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:13.231 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:14:13.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T16:14:13.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:13.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:14:13.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:14.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:14.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:14.663 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:14:14.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T16:14:14.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:14.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:14:15.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:16.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:16.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:16.507 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:14:16.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T16:14:16.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:16.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:14:16.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:17.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:17.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:17.853 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:14:17.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T16:14:17.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:17.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:14:18.188 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3301932794,v1:127.0.0.1:6819/3301932794] [v2:127.0.0.1:6820/3301932794,v1:127.0.0.1:6821/3301932794] exists,up 1e92deb4-fd93-4ca3-b1a0-c3adba3258c0 2024-09-13T16:14:18.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:14:18.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:14:18.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:14:18.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:665: TEST_pg_dump_objects_scrubbed: create_pool test 1 1 2024-09-13T16:14:18.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-13T16:14:18.674 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'test' created 2024-09-13T16:14:18.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T16:14:19.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:666: TEST_pg_dump_objects_scrubbed: wait_for_clean 2024-09-13T16:14:19.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:14:19.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:14:19.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:14:19.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:14:19.696 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:14:19.696 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:14:19.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:14:19.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:14:19.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:14:19.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:14:19.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:14:19.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:14:19.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:14:19.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:14:19.823 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:14:20.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:14:20.174 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:14:20.174 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:14:20.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:14:20.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:14:20.174 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:14:20.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-13T16:14:20.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-13T16:14:20.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-13T16:14:20.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:14:20.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:14:20.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672980 2024-09-13T16:14:20.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672980 2024-09-13T16:14:20.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672980' 2024-09-13T16:14:20.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:14:20.460 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:14:20.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509444 2024-09-13T16:14:20.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509444 2024-09-13T16:14:20.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672980 2-64424509444' 2024-09-13T16:14:20.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:14:20.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-13T16:14:20.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:14:20.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:14:20.596 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-13T16:14:20.596 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:14:20.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-13T16:14:20.597 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836502 2024-09-13T16:14:20.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-13T16:14:20.597 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:14:21.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-13T16:14:21.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:14:21.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672980 2024-09-13T16:14:21.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:14:21.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:14:21.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672980 2024-09-13T16:14:21.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:14:21.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672980 2024-09-13T16:14:21.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-13T16:14:21.027 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672980 2024-09-13T16:14:21.027 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:14:21.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-13T16:14:21.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:14:21.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509444 2024-09-13T16:14:21.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:14:21.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:14:21.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509444 2024-09-13T16:14:21.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:14:21.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509444 2024-09-13T16:14:21.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-09-13T16:14:21.358 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509444 2024-09-13T16:14:21.359 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:14:21.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509444 -lt 64424509444 2024-09-13T16:14:21.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:14:21.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:14:21.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:14:22.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:14:22.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:14:22.165 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:14:22.165 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:14:22.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:14:22.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:14:22.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:14:22.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:14:22.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T16:14:22.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:14:22.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:14:22.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:14:22.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T16:14:22.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T16:14:22.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T16:14:22.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T16:14:22.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T16:14:22.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T16:14:22.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T16:14:22.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:14:22.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:14:22.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:14:22.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:14:22.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:14:22.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:14:22.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:14:23.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T16:14:23.285 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:14:23.286 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:14:23.286 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:14:23.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T16:14:23.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T16:14:23.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T16:14:23.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T16:14:23.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T16:14:23.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T16:14:23.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T16:14:23.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T16:14:23.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T16:14:24.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T16:14:24.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T16:14:24.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T16:14:24.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T16:14:24.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T16:14:24.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T16:14:24.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:14:24.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:14:24.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:14:24.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:14:24.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:14:24.302 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:14:24.302 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:14:24.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T16:14:24.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:14:24.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:14:24.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:14:24.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T16:14:24.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T16:14:24.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T16:14:24.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T16:14:24.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T16:14:24.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T16:14:24.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T16:14:24.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T16:14:24.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T16:14:25.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T16:14:25.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T16:14:25.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-13T16:14:25.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T16:14:25.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-13T16:14:25.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T16:14:25.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:14:25.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:14:25.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:14:25.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:14:25.812 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:14:25.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:14:25.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:14:26.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T16:14:26.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:14:26.130 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:14:26.130 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:14:26.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T16:14:26.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T16:14:26.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T16:14:26.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T16:14:26.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T16:14:26.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T16:14:26.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T16:14:26.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T16:14:26.517 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T16:14:26.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T16:14:26.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T16:14:26.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 13 )) 2024-09-13T16:14:26.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T16:14:26.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-13T16:14:27.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T16:14:27.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:14:27.800 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:14:27.800 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:14:27.800 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:14:27.800 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:14:27.800 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:14:27.800 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:14:28.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:14:28.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:14:28.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:14:28.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:14:28.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:14:28.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:14:28.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:14:28.502 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:667: TEST_pg_dump_objects_scrubbed: ceph osd dump 2024-09-13T16:14:28.502 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:667: TEST_pg_dump_objects_scrubbed: grep '^pool.*['\'']test['\'']' 2024-09-13T16:14:28.503 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:667: TEST_pg_dump_objects_scrubbed: awk '{ print $2 }' 2024-09-13T16:14:28.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:667: TEST_pg_dump_objects_scrubbed: poolid=1 2024-09-13T16:14:28.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:669: TEST_pg_dump_objects_scrubbed: dd if=/dev/urandom of=testdata.528094 bs=1032 count=1 2024-09-13T16:14:28.865 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T16:14:28.865 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T16:14:28.865 INFO:tasks.workunit.client.0.smithi112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000107695 s, 9.6 MB/s 2024-09-13T16:14:28.866 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: seq 1 15 2024-09-13T16:14:28.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-13T16:14:28.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: rados -p test put obj1 testdata.528094 2024-09-13T16:14:29.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-13T16:14:29.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: rados -p test put obj2 testdata.528094 2024-09-13T16:14:29.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-13T16:14:29.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: rados -p test put obj3 testdata.528094 2024-09-13T16:14:29.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-13T16:14:29.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: rados -p test put obj4 testdata.528094 2024-09-13T16:14:29.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-13T16:14:29.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: rados -p test put obj5 testdata.528094 2024-09-13T16:14:29.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-13T16:14:29.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: rados -p test put obj6 testdata.528094 2024-09-13T16:14:30.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-13T16:14:30.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: rados -p test put obj7 testdata.528094 2024-09-13T16:14:30.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-13T16:14:30.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: rados -p test put obj8 testdata.528094 2024-09-13T16:14:30.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-13T16:14:30.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: rados -p test put obj9 testdata.528094 2024-09-13T16:14:30.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-13T16:14:30.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: rados -p test put obj10 testdata.528094 2024-09-13T16:14:31.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-13T16:14:31.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: rados -p test put obj11 testdata.528094 2024-09-13T16:14:31.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-13T16:14:31.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: rados -p test put obj12 testdata.528094 2024-09-13T16:14:31.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-13T16:14:31.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: rados -p test put obj13 testdata.528094 2024-09-13T16:14:31.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-13T16:14:31.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: rados -p test put obj14 testdata.528094 2024-09-13T16:14:31.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2024-09-13T16:14:31.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: rados -p test put obj15 testdata.528094 2024-09-13T16:14:32.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:674: TEST_pg_dump_objects_scrubbed: rm -f testdata.528094 2024-09-13T16:14:32.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:676: TEST_pg_dump_objects_scrubbed: local pgid=1.0 2024-09-13T16:14:32.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:678: TEST_pg_dump_objects_scrubbed: pg_scrub 1.0 2024-09-13T16:14:32.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=1.0 2024-09-13T16:14:32.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 1.0 2024-09-13T16:14:32.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:14:32.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:14:32.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:14:32.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:14:32.478 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-13T16:14:18.604431+0000 2024-09-13T16:14:32.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 1.0 2024-09-13T16:14:32.778 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.0 on osd.1 to scrub 2024-09-13T16:14:32.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 1.0 2024-09-13T16:14:18.604431+0000 2024-09-13T16:14:32.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T16:14:32.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T16:14:18.604431+0000 2024-09-13T16:14:32.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T16:14:32.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T16:14:32.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:14:32.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:14:32.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:14:32.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:14:32.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:14:32.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:14:33.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:14:18.604431+0000 '>' 2024-09-13T16:14:18.604431+0000 2024-09-13T16:14:33.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:14:34.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:14:34.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:14:34.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:14:34.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:14:34.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:14:34.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:14:34.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:14:34.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:14:18.604431+0000 '>' 2024-09-13T16:14:18.604431+0000 2024-09-13T16:14:34.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:14:35.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:14:35.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:14:35.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:14:35.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:14:35.438 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:14:35.438 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:14:35.438 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:14:35.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:14:18.604431+0000 '>' 2024-09-13T16:14:18.604431+0000 2024-09-13T16:14:35.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:14:36.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:14:36.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:14:36.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:14:36.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:14:36.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:14:36.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:14:36.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:14:37.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:14:33.779586+0000 '>' 2024-09-13T16:14:18.604431+0000 2024-09-13T16:14:37.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T16:14:37.084 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:679: TEST_pg_dump_objects_scrubbed: ceph pg 1.0 query 2024-09-13T16:14:37.084 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:679: TEST_pg_dump_objects_scrubbed: jq .info.stats.objects_scrubbed 2024-09-13T16:14:37.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:679: TEST_pg_dump_objects_scrubbed: test 15 = 15 2024-09-13T16:14:37.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:680: TEST_pg_dump_objects_scrubbed: perf_counters td/osd-scrub-test 3 2024-09-13T16:14:37.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-09-13T16:14:37.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2024-09-13T16:14:37.230 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2024-09-13T16:14:37.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2024-09-13T16:14:37.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:14:37.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-09-13T16:14:37.232 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:14:37.353 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:14:37.354 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:14:37.354 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:14:37.354 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:14:37.354 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:14:37.354 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:14:37.354 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.354 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:14:37.354 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:14:37.354 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:14:37.354 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:14:37.354 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:14:37.354 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.355 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.355 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.355 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.355 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:14:37.355 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:14:37.355 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.355 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.355 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.355 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.355 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:14:37.355 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:14:37.355 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:14:37.356 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:14:37.356 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:14:37.356 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:14:37.356 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:14:37.356 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.356 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.356 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.356 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.356 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:14:37.356 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:14:37.356 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:14:37.357 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:14:37.357 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.357 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.357 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.357 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.357 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:14:37.357 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.357 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.357 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:14:37.357 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:14:37.357 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:14:37.357 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:14:37.358 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:14:37.358 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:14:37.358 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.358 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:14:37.358 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:14:37.358 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:14:37.358 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:14:37.358 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:14:37.358 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.358 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.358 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.358 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.358 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:14:37.359 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:14:37.359 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.359 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.359 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.359 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.359 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:14:37.359 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:14:37.359 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:14:37.359 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:14:37.359 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:14:37.359 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:14:37.359 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:14:37.360 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.360 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.360 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.360 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.360 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:14:37.360 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:14:37.360 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:14:37.360 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:14:37.360 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.360 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.360 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.360 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.361 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:14:37.361 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.361 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.361 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:14:37.361 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:14:37.361 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:14:37.361 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:14:37.361 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:14:37.361 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:14:37.361 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.361 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:14:37.361 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:14:37.361 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:14:37.362 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:14:37.362 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:14:37.362 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.362 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.362 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.362 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.362 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:14:37.362 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:14:37.362 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.362 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.362 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.362 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.362 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:14:37.362 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:14:37.363 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:14:37.363 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:14:37.363 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:14:37.363 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:14:37.363 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:14:37.363 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.363 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.363 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.363 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.363 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:14:37.363 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:14:37.363 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:14:37.363 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:14:37.363 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.364 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.364 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.364 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.364 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:14:37.364 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.364 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.364 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:14:37.364 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:14:37.364 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:14:37.364 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:14:37.364 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:14:37.364 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:14:37.364 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.365 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:14:37.365 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:14:37.365 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:14:37.365 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:14:37.365 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:14:37.365 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.365 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.365 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.365 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.365 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:14:37.365 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:14:37.365 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.365 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.365 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.366 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.366 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:14:37.366 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:14:37.366 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:14:37.366 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:14:37.366 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:14:37.366 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:14:37.366 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:14:37.366 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.367 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.367 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.367 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.367 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:14:37.367 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:14:37.367 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:14:37.367 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:14:37.367 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.367 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.367 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.367 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.367 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:14:37.367 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.368 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.368 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:14:37.368 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:14:37.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:14:37.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-09-13T16:14:37.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:14:37.489 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:14:37.489 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:14:37.489 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:14:37.489 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:14:37.489 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:14:37.490 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:14:37.490 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.490 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:14:37.490 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:14:37.490 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:14:37.490 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:14:37.490 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:14:37.490 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.490 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.490 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.490 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.490 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:14:37.490 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:14:37.491 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.491 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.491 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.491 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.491 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:14:37.491 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:14:37.491 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:14:37.491 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:14:37.491 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:14:37.491 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:14:37.491 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:14:37.491 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.491 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.491 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.492 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.492 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:14:37.492 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:14:37.492 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:14:37.492 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:14:37.492 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.492 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.492 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.492 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.492 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:14:37.492 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.492 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.493 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:14:37.493 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:14:37.493 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:14:37.493 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:14:37.493 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:14:37.493 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:14:37.493 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.493 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:14:37.493 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:14:37.493 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:14:37.493 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:14:37.493 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:14:37.493 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.494 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.494 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.494 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.494 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:14:37.494 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:14:37.494 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.494 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.494 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.494 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.494 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:14:37.494 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:14:37.494 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:14:37.494 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:14:37.494 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:14:37.495 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:14:37.495 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:14:37.495 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.495 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.495 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.495 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.495 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:14:37.495 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:14:37.495 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:14:37.495 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:14:37.495 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.495 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.495 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.496 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.496 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:14:37.496 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.496 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.496 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:14:37.496 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:14:37.496 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:14:37.496 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:14:37.496 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:14:37.496 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:14:37.496 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.496 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:14:37.496 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:14:37.496 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:14:37.497 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:14:37.497 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:14:37.497 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.497 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.497 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.497 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.497 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:14:37.497 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:14:37.497 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.497 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.497 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.497 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.497 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:14:37.497 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:14:37.498 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:14:37.498 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:14:37.498 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:14:37.498 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:14:37.498 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:14:37.498 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.498 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.498 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.498 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.498 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:14:37.498 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:14:37.498 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:14:37.498 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:14:37.498 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.499 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.499 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.499 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.499 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:14:37.499 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.499 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.499 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:14:37.499 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:14:37.499 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:14:37.499 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:14:37.499 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:14:37.499 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:14:37.499 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.499 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:14:37.500 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 1, 2024-09-13T16:14:37.500 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 1, 2024-09-13T16:14:37.500 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 1, 2024-09-13T16:14:37.500 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:14:37.500 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 1, 2024-09-13T16:14:37.500 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.500 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.500 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.500 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:14:37.500 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:14:37.500 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.500 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.500 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.500 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.501 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:14:37.501 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 1, 2024-09-13T16:14:37.501 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:14:37.501 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:14:37.501 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:14:37.501 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:14:37.501 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:14:37.501 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.501 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.502 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.502 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.502 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:14:37.502 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:14:37.502 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 1, 2024-09-13T16:14:37.502 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:14:37.502 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.502 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.502 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.502 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.502 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 2 2024-09-13T16:14:37.502 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.502 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.502 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:14:37.503 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:14:37.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:14:37.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2024-09-13T16:14:37.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:14:37.627 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:14:37.627 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:14:37.627 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:14:37.627 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:14:37.627 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:14:37.627 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:14:37.627 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.627 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:14:37.627 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:14:37.627 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:14:37.627 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:14:37.628 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:14:37.628 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.628 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.628 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.628 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.628 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:14:37.628 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:14:37.628 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.628 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.628 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.628 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.628 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:14:37.628 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:14:37.629 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:14:37.629 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:14:37.629 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:14:37.629 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:14:37.629 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:14:37.629 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.629 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.629 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.629 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.629 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:14:37.629 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:14:37.629 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:14:37.629 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:14:37.630 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.630 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.630 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.630 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.630 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:14:37.630 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.630 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.630 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:14:37.630 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:14:37.630 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:14:37.630 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:14:37.630 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:14:37.630 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:14:37.630 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.631 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:14:37.631 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:14:37.631 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:14:37.631 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:14:37.631 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:14:37.631 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.631 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.631 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.631 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.631 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:14:37.631 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:14:37.631 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.631 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.631 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.632 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.632 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:14:37.632 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:14:37.632 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:14:37.632 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:14:37.632 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:14:37.632 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:14:37.632 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:14:37.632 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.632 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.632 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.632 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.632 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:14:37.632 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:14:37.632 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:14:37.633 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:14:37.633 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.633 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.633 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.633 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.633 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:14:37.633 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.633 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.633 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:14:37.633 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:14:37.633 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:14:37.633 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:14:37.633 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:14:37.633 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:14:37.634 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.634 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:14:37.634 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:14:37.634 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:14:37.634 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:14:37.634 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:14:37.634 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.634 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.634 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.634 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.634 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:14:37.634 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:14:37.634 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.635 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.635 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.635 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.635 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:14:37.635 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:14:37.635 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:14:37.635 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:14:37.635 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:14:37.635 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:14:37.635 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:14:37.635 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.635 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.635 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.635 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.636 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:14:37.636 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:14:37.636 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:14:37.636 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:14:37.636 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.636 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.636 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.636 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.636 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:14:37.636 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.636 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.636 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:14:37.636 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:14:37.636 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:14:37.637 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:14:37.637 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:14:37.637 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:14:37.637 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.637 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:14:37.637 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:14:37.637 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:14:37.637 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:14:37.637 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:14:37.637 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.637 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.637 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.637 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.637 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:14:37.637 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:14:37.638 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.638 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.638 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.638 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.638 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:14:37.638 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:14:37.638 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:14:37.638 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:14:37.638 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:14:37.638 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:14:37.638 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:14:37.639 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.639 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.639 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.639 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.639 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:14:37.639 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:14:37.639 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:14:37.639 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:14:37.639 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:14:37.639 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:14:37.640 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:14:37.640 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:14:37.640 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:14:37.640 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.640 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:14:37.640 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:14:37.640 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:14:37.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:682: TEST_pg_dump_objects_scrubbed: teardown td/osd-scrub-test 2024-09-13T16:14:37.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:14:37.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:14:37.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:14:37.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:14:37.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:14:37.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:14:37.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:14:37.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:14:37.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:14:37.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:14:37.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:14:37.952 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:14:37.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:14:37.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:14:37.954 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:14:37.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:14:37.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:14:37.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:14:37.956 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:14:37.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:14:37.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:14:37.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:14:37.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:14:37.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:14:37.979 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:14:37.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:14:37.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:14:37.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:14:37.981 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Teardown Test TEST_pg_dump_objects_scrubbed ------------------ 2024-09-13T16:14:37.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_pg_dump_objects_scrubbed ------------------' 2024-09-13T16:14:37.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-09-13T16:14:37.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:14:37.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:14:37.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:14:37.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:14:37.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:14:37.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:14:37.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:14:37.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:14:37.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:14:37.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:14:37.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:14:37.987 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:14:37.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:14:37.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:14:37.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:14:37.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:14:37.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:14:37.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:14:37.990 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:14:37.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:14:37.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:14:37.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:14:37.993 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:14:37.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:14:37.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:14:37.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:14:37.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:14:37.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:14:37.996 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Complete Test TEST_pg_dump_objects_scrubbed ------------------ 2024-09-13T16:14:37.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_pg_dump_objects_scrubbed ------------------' 2024-09-13T16:14:37.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-13T16:14:37.997 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Prepare Test TEST_scrub_abort ------------------- 2024-09-13T16:14:37.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_scrub_abort -------------------' 2024-09-13T16:14:37.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-09-13T16:14:37.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2024-09-13T16:14:37.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2024-09-13T16:14:37.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:14:37.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:14:37.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:14:37.999 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:14:37.999 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:14:37.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:14:37.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:14:37.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:14:37.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:14:37.999 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:14:37.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:14:38.000 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:14:38.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:14:38.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:14:38.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:14:38.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:14:38.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:14:38.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:14:38.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:14:38.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:14:38.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:14:38.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:14:38.006 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:14:38.007 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:14:38.007 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:14:38.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:14:38.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:14:38.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:14:38.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2024-09-13T16:14:38.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T16:14:38.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:14:38.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:14:38.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.528094 2024-09-13T16:14:38.011 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T16:14:38.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T16:14:38.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T16:14:38.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-13T16:14:38.012 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Run Test TEST_scrub_abort ----------------------- 2024-09-13T16:14:38.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_scrub_abort -----------------------' 2024-09-13T16:14:38.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_scrub_abort td/osd-scrub-test 2024-09-13T16:14:38.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:411: TEST_scrub_abort: local dir=td/osd-scrub-test 2024-09-13T16:14:38.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:412: TEST_scrub_abort: _scrub_abort td/osd-scrub-test scrub 2024-09-13T16:14:38.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: local dir=td/osd-scrub-test 2024-09-13T16:14:38.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:301: _scrub_abort: local poolname=test 2024-09-13T16:14:38.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:302: _scrub_abort: local OSDS=3 2024-09-13T16:14:38.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:303: _scrub_abort: local objects=1000 2024-09-13T16:14:38.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:304: _scrub_abort: local type=scrub 2024-09-13T16:14:38.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:306: _scrub_abort: TESTDATA=testdata.528094 2024-09-13T16:14:38.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:307: _scrub_abort: test scrub = scrub 2024-09-13T16:14:38.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:309: _scrub_abort: stopscrub=noscrub 2024-09-13T16:14:38.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: check=noscrub 2024-09-13T16:14:38.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-13T16:14:38.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2024-09-13T16:14:38.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T16:14:38.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T16:14:38.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T16:14:38.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2024-09-13T16:14:38.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-13T16:14:38.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T16:14:38.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:14:38.489 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:14:38.489 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:14:38.489 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:14:38.489 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:14:38.490 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:14:38.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-13T16:14:38.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T16:14:38.752 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T16:14:38.752 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T16:14:38.752 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T16:14:38.752 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T16:14:38.753 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T16:14:38.753 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T16:14:38.753 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T16:14:38.753 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T16:14:38.753 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:14:38.754 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:14:38.754 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:14:38.754 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-mon.a.asok 2024-09-13T16:14:38.754 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T16:14:38.754 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.528094/ceph-mon.a.asok config get fsid 2024-09-13T16:14:38.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T16:14:38.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T16:14:38.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T16:14:38.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T16:14:38.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T16:14:38.876 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T16:14:38.876 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T16:14:38.876 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T16:14:38.876 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:14:38.876 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:14:38.876 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:14:38.876 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-mon.a.asok 2024-09-13T16:14:38.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T16:14:38.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.528094/ceph-mon.a.asok config get mon_host 2024-09-13T16:14:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_mgr td/osd-scrub-test x 2024-09-13T16:14:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2024-09-13T16:14:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T16:14:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T16:14:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T16:14:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2024-09-13T16:14:38.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T16:14:39.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T16:14:39.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:14:39.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:14:39.913 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:14:39.913 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:14:39.913 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:14:39.913 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:14:39.914 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T16:14:39.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T16:14:39.951 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: expr 3 - 1 2024-09-13T16:14:39.951 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: seq 0 2 2024-09-13T16:14:39.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:14:39.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:324: _scrub_abort: run_osd td/osd-scrub-test 0 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-13T16:14:39.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:14:39.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:14:39.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T16:14:39.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:14:39.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-13T16:14:39.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:14:39.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:14:39.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:14:39.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:14:39.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-13T16:14:39.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-13T16:14:39.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:14:39.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:14:39.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:14:39.956 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:14:39.956 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:14:39.956 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:14:39.956 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:14:39.956 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:14:39.957 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:14:39.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:14:39.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:14:39.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:14:39.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:14:39.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:14:39.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:14:39.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:14:39.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:14:39.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:14:39.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:14:39.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:14:39.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:14:39.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-13T16:14:39.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-13T16:14:39.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:14:39.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0489a71d-4b61-4394-81ce-09f842dd4a13 2024-09-13T16:14:39.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0489a71d-4b61-4394-81ce-09f842dd4a13' 2024-09-13T16:14:39.959 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 0489a71d-4b61-4394-81ce-09f842dd4a13 2024-09-13T16:14:39.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:14:39.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDvZORmfN4cOhAAQ+ej4evWlv2ncjRVmfZrOQ== 2024-09-13T16:14:39.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDvZORmfN4cOhAAQ+ej4evWlv2ncjRVmfZrOQ=="}' 2024-09-13T16:14:39.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0489a71d-4b61-4394-81ce-09f842dd4a13 -i td/osd-scrub-test/0/new.json 2024-09-13T16:14:40.270 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:14:40.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-13T16:14:40.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQDvZORmfN4cOhAAQ+ej4evWlv2ncjRVmfZrOQ== --osd-uuid 0489a71d-4b61-4394-81ce-09f842dd4a13 2024-09-13T16:14:40.318 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:14:40.312+0000 7fab140477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:14:40.380 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:14:40.376+0000 7fab140477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:14:40.382 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:14:40.376+0000 7fab140477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:14:40.382 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:14:40.380+0000 7fab140477c0 -1 bdev(0x5582ba6f8800 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2024-09-13T16:14:40.383 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:14:40.380+0000 7fab140477c0 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-13T16:14:48.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-13T16:14:48.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:14:48.604 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T16:14:48.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T16:14:48.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:14:49.002 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T16:14:49.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T16:14:49.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:14:49.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:14:49.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:14:49.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-13T16:14:49.036 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:14:49.032+0000 7f74f4ab17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:14:49.086 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:14:49.080+0000 7f74f4ab17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:14:49.088 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:14:49.084+0000 7f74f4ab17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:14:49.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T16:14:49.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:14:49.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T16:14:49.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:14:49.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:14:49.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:49.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:14:49.308 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:14:49.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:49.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:14:49.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:50.161 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:14:50.156+0000 7f74f4ab17c0 -1 Falling back to public interface 2024-09-13T16:14:50.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:50.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:50.602 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:14:50.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:14:50.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:50.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:14:50.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:51.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:51.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:51.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:14:51.919 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:14:51.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:51.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:14:52.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:53.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:53.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:53.234 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:14:53.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:14:53.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:53.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:14:53.428 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:14:53.424+0000 7f74f4ab17c0 -1 osd.0 0 log_to_monitors true 2024-09-13T16:14:53.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:14:54.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:14:54.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:14:54.584 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:14:54.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:14:54.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:14:54.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:14:54.929 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1134512125,v1:127.0.0.1:6803/1134512125] [v2:127.0.0.1:6804/1134512125,v1:127.0.0.1:6805/1134512125] exists,up 0489a71d-4b61-4394-81ce-09f842dd4a13 2024-09-13T16:14:54.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:14:54.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:14:54.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:14:54.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:14:54.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:324: _scrub_abort: run_osd td/osd-scrub-test 1 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-13T16:14:54.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:14:54.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:14:54.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T16:14:54.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:14:54.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-13T16:14:54.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:14:54.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:14:54.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:14:54.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:14:54.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-13T16:14:54.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-13T16:14:54.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:14:54.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:14:54.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:14:54.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:14:54.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:14:54.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:14:54.934 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:14:54.934 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:14:54.934 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:14:54.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:14:54.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:14:54.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:14:54.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:14:54.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:14:54.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:14:54.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:14:54.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:14:54.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:14:54.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:14:54.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:14:54.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:14:54.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-13T16:14:54.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-13T16:14:54.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:14:54.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ad9c4ba9-15bc-425d-9450-c587dc6bd25d 2024-09-13T16:14:54.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ad9c4ba9-15bc-425d-9450-c587dc6bd25d' 2024-09-13T16:14:54.937 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 ad9c4ba9-15bc-425d-9450-c587dc6bd25d 2024-09-13T16:14:54.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:14:54.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD+ZORmg5e8OBAAFxo/pOraUihAM4qAwbmthw== 2024-09-13T16:14:54.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD+ZORmg5e8OBAAFxo/pOraUihAM4qAwbmthw=="}' 2024-09-13T16:14:54.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ad9c4ba9-15bc-425d-9450-c587dc6bd25d -i td/osd-scrub-test/1/new.json 2024-09-13T16:14:55.446 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:14:55.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-13T16:14:55.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQD+ZORmg5e8OBAAFxo/pOraUihAM4qAwbmthw== --osd-uuid ad9c4ba9-15bc-425d-9450-c587dc6bd25d 2024-09-13T16:14:55.504 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:14:55.504+0000 7f8de0c087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:14:55.583 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:14:55.580+0000 7f8de0c087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:14:55.584 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:14:55.584+0000 7f8de0c087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:14:55.584 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:14:55.584+0000 7f8de0c087c0 -1 bdev(0x5596164bdc00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2024-09-13T16:14:55.584 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:14:55.584+0000 7f8de0c087c0 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-13T16:15:04.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-13T16:15:04.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:15:04.227 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T16:15:04.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T16:15:04.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:15:04.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T16:15:04.635 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T16:15:04.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:15:04.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:15:04.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-13T16:15:04.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:15:04.671 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:15:04.672+0000 7f1beda127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:15:04.704 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:15:04.704+0000 7f1beda127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:15:04.705 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:15:04.704+0000 7f1beda127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:15:04.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T16:15:04.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:15:04.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T16:15:04.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:15:04.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:15:04.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:15:04.955 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:15:04.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:15:04.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:15:04.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:15:05.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:15:06.230 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:15:06.228+0000 7f1beda127c0 -1 Falling back to public interface 2024-09-13T16:15:06.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:15:06.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:15:06.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:15:06.335 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:15:06.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:15:06.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:15:06.666 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:15:07.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:15:07.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:15:07.668 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:15:07.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:15:07.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:15:07.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:15:07.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:15:08.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:15:08.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:15:08.995 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:15:08.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:15:08.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:15:08.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:15:09.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:15:09.695 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:15:09.696+0000 7f1beda127c0 -1 osd.1 0 log_to_monitors true 2024-09-13T16:15:10.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:15:10.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:15:10.330 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:15:10.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:15:10.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:15:10.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:15:10.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:15:11.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:15:11.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:15:11.753 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:15:11.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:15:11.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:15:11.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:15:12.103 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1075556937,v1:127.0.0.1:6811/1075556937] [v2:127.0.0.1:6812/1075556937,v1:127.0.0.1:6813/1075556937] exists,up ad9c4ba9-15bc-425d-9450-c587dc6bd25d 2024-09-13T16:15:12.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:15:12.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:15:12.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:15:12.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:318: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:15:12.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:324: _scrub_abort: run_osd td/osd-scrub-test 2 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-13T16:15:12.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:15:12.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:15:12.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T16:15:12.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:15:12.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-13T16:15:12.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:15:12.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:15:12.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:15:12.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:15:12.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-13T16:15:12.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-13T16:15:12.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:15:12.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:15:12.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:15:12.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:15:12.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:15:12.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:15:12.108 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:15:12.108 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:15:12.108 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:15:12.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:15:12.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:15:12.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:15:12.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:15:12.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:15:12.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:15:12.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:15:12.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:15:12.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:15:12.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:15:12.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:15:12.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:15:12.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-09-13T16:15:12.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-13T16:15:12.110 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:15:12.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=33a26560-9e5d-41eb-a9a0-61dc0d9c39f3 2024-09-13T16:15:12.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 33a26560-9e5d-41eb-a9a0-61dc0d9c39f3' 2024-09-13T16:15:12.110 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 33a26560-9e5d-41eb-a9a0-61dc0d9c39f3 2024-09-13T16:15:12.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:15:12.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAQZeRmuXKtBxAA2QwjQYuzSqka7giMQnsI5Q== 2024-09-13T16:15:12.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAQZeRmuXKtBxAA2QwjQYuzSqka7giMQnsI5Q=="}' 2024-09-13T16:15:12.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 33a26560-9e5d-41eb-a9a0-61dc0d9c39f3 -i td/osd-scrub-test/2/new.json 2024-09-13T16:15:12.600 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:15:12.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-13T16:15:12.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQAQZeRmuXKtBxAA2QwjQYuzSqka7giMQnsI5Q== --osd-uuid 33a26560-9e5d-41eb-a9a0-61dc0d9c39f3 2024-09-13T16:15:12.657 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:15:12.656+0000 7f416bf147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:15:12.788 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:15:12.788+0000 7f416bf147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:15:12.789 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:15:12.788+0000 7f416bf147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:15:12.790 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:15:12.788+0000 7f416bf147c0 -1 bdev(0x5576be00bc00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2024-09-13T16:15:12.790 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:15:12.788+0000 7f416bf147c0 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-13T16:15:21.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-13T16:15:21.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:15:21.051 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T16:15:21.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T16:15:21.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:15:21.469 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T16:15:21.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T16:15:21.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-09-13T16:15:21.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:15:21.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:15:21.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:15:21.504 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:15:21.503+0000 7fd13ef8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:15:21.551 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:15:21.551+0000 7fd13ef8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:15:21.553 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:15:21.551+0000 7fd13ef8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:15:21.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T16:15:21.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:15:21.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T16:15:21.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:15:21.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:15:21.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:15:21.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:15:21.793 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:15:21.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:15:21.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:15:22.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:15:22.857 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:15:22.855+0000 7fd13ef8a7c0 -1 Falling back to public interface 2024-09-13T16:15:23.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:15:23.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:15:23.121 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:15:23.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:15:23.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:15:23.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:15:23.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:15:24.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:15:24.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:15:24.442 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:15:24.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:15:24.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:15:24.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:15:24.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:15:25.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:15:25.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:15:25.778 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:15:25.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:15:25.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:15:25.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:15:26.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:15:26.339 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:15:26.339+0000 7fd13ef8a7c0 -1 osd.2 0 log_to_monitors true 2024-09-13T16:15:27.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:15:27.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:15:27.091 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:15:27.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:15:27.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:15:27.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:15:27.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:15:28.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:15:28.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:15:28.590 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:15:28.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:15:28.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:15:28.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:15:28.947 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1429155543,v1:127.0.0.1:6819/1429155543] [v2:127.0.0.1:6820/1429155543,v1:127.0.0.1:6821/1429155543] exists,up 33a26560-9e5d-41eb-a9a0-61dc0d9c39f3 2024-09-13T16:15:28.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:15:28.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:15:28.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:15:28.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: create_pool test 1 1 2024-09-13T16:15:28.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-13T16:15:29.483 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'test' created 2024-09-13T16:15:29.505 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T16:15:30.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:333: _scrub_abort: wait_for_clean 2024-09-13T16:15:30.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:15:30.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:15:30.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:15:30.508 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:15:30.509 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:15:30.509 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:15:30.510 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:15:30.510 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:15:30.510 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:15:30.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:15:30.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:15:30.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:15:30.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:15:30.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:15:30.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:15:30.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:15:30.972 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:15:30.972 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:15:30.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:15:30.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:15:30.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:15:31.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836489 2024-09-13T16:15:31.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836489 2024-09-13T16:15:31.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489' 2024-09-13T16:15:31.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:15:31.313 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:15:31.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-13T16:15:31.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-13T16:15:31.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489 1-42949672965' 2024-09-13T16:15:31.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:15:31.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:15:31.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509442 2024-09-13T16:15:31.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509442 2024-09-13T16:15:31.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489 1-42949672965 2-64424509442' 2024-09-13T16:15:31.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:15:31.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836489 2024-09-13T16:15:31.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:15:31.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:15:31.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836489 2024-09-13T16:15:31.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:15:31.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836489 2024-09-13T16:15:31.626 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836489 2024-09-13T16:15:31.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-09-13T16:15:31.626 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:15:31.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836489 2024-09-13T16:15:31.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:15:32.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:15:32.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:15:33.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836489 2024-09-13T16:15:33.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:15:33.295 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-13T16:15:33.295 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:15:33.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:15:33.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-13T16:15:33.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:15:33.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-13T16:15:33.299 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672965 2024-09-13T16:15:33.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-13T16:15:33.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:15:33.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672966 -lt 42949672965 2024-09-13T16:15:33.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:15:33.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509442 2024-09-13T16:15:33.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:15:33.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:15:33.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509442 2024-09-13T16:15:33.627 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:15:33.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509442 2024-09-13T16:15:33.628 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509442 2024-09-13T16:15:33.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-09-13T16:15:33.629 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:15:34.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509442 -lt 64424509442 2024-09-13T16:15:34.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:15:34.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:15:34.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:15:34.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:15:34.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:15:34.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:15:34.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:15:34.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:15:34.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:15:34.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:15:34.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:15:34.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:15:34.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:15:34.706 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:15:34.706 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:15:35.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:15:35.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:15:35.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:15:35.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: ceph osd dump 2024-09-13T16:15:35.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: grep '^pool.*['\'']test['\'']' 2024-09-13T16:15:35.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: awk '{ print $2 }' 2024-09-13T16:15:35.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: poolid=1 2024-09-13T16:15:35.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: dd if=/dev/urandom of=testdata.528094 bs=1032 count=1 2024-09-13T16:15:35.389 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T16:15:35.389 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T16:15:35.389 INFO:tasks.workunit.client.0.smithi112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.000116783 s, 8.8 MB/s 2024-09-13T16:15:35.390 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: seq 1 1000 2024-09-13T16:15:35.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:35.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj1 testdata.528094 2024-09-13T16:15:35.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:35.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj2 testdata.528094 2024-09-13T16:15:35.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:35.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj3 testdata.528094 2024-09-13T16:15:35.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:35.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj4 testdata.528094 2024-09-13T16:15:36.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:36.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj5 testdata.528094 2024-09-13T16:15:36.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:36.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj6 testdata.528094 2024-09-13T16:15:36.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:36.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj7 testdata.528094 2024-09-13T16:15:36.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:36.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj8 testdata.528094 2024-09-13T16:15:37.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:37.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj9 testdata.528094 2024-09-13T16:15:37.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:37.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj10 testdata.528094 2024-09-13T16:15:37.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:37.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj11 testdata.528094 2024-09-13T16:15:37.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:37.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj12 testdata.528094 2024-09-13T16:15:37.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:37.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj13 testdata.528094 2024-09-13T16:15:38.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:38.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj14 testdata.528094 2024-09-13T16:15:38.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:38.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj15 testdata.528094 2024-09-13T16:15:38.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:38.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj16 testdata.528094 2024-09-13T16:15:38.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:38.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj17 testdata.528094 2024-09-13T16:15:38.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:38.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj18 testdata.528094 2024-09-13T16:15:39.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:39.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj19 testdata.528094 2024-09-13T16:15:39.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:39.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj20 testdata.528094 2024-09-13T16:15:39.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:39.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj21 testdata.528094 2024-09-13T16:15:39.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:39.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj22 testdata.528094 2024-09-13T16:15:40.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:40.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj23 testdata.528094 2024-09-13T16:15:40.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:40.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj24 testdata.528094 2024-09-13T16:15:40.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:40.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj25 testdata.528094 2024-09-13T16:15:40.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:40.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj26 testdata.528094 2024-09-13T16:15:40.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:40.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj27 testdata.528094 2024-09-13T16:15:41.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:41.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj28 testdata.528094 2024-09-13T16:15:41.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:41.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj29 testdata.528094 2024-09-13T16:15:41.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:41.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj30 testdata.528094 2024-09-13T16:15:41.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:41.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj31 testdata.528094 2024-09-13T16:15:41.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:41.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj32 testdata.528094 2024-09-13T16:15:42.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:42.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj33 testdata.528094 2024-09-13T16:15:42.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:42.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj34 testdata.528094 2024-09-13T16:15:42.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:42.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj35 testdata.528094 2024-09-13T16:15:42.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:42.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj36 testdata.528094 2024-09-13T16:15:42.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:42.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj37 testdata.528094 2024-09-13T16:15:43.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:43.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj38 testdata.528094 2024-09-13T16:15:43.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:43.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj39 testdata.528094 2024-09-13T16:15:43.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:43.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj40 testdata.528094 2024-09-13T16:15:43.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:43.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj41 testdata.528094 2024-09-13T16:15:44.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:44.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj42 testdata.528094 2024-09-13T16:15:44.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:44.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj43 testdata.528094 2024-09-13T16:15:44.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:44.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj44 testdata.528094 2024-09-13T16:15:44.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:44.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj45 testdata.528094 2024-09-13T16:15:44.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:44.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj46 testdata.528094 2024-09-13T16:15:45.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:45.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj47 testdata.528094 2024-09-13T16:15:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:45.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj48 testdata.528094 2024-09-13T16:15:45.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:45.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj49 testdata.528094 2024-09-13T16:15:45.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:45.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj50 testdata.528094 2024-09-13T16:15:45.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:45.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj51 testdata.528094 2024-09-13T16:15:46.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:46.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj52 testdata.528094 2024-09-13T16:15:46.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:46.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj53 testdata.528094 2024-09-13T16:15:46.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:46.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj54 testdata.528094 2024-09-13T16:15:46.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:46.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj55 testdata.528094 2024-09-13T16:15:46.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:46.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj56 testdata.528094 2024-09-13T16:15:47.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:47.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj57 testdata.528094 2024-09-13T16:15:47.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:47.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj58 testdata.528094 2024-09-13T16:15:47.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:47.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj59 testdata.528094 2024-09-13T16:15:47.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:47.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj60 testdata.528094 2024-09-13T16:15:48.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:48.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj61 testdata.528094 2024-09-13T16:15:48.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:48.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj62 testdata.528094 2024-09-13T16:15:48.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:48.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj63 testdata.528094 2024-09-13T16:15:48.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:48.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj64 testdata.528094 2024-09-13T16:15:48.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:48.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj65 testdata.528094 2024-09-13T16:15:49.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:49.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj66 testdata.528094 2024-09-13T16:15:49.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:49.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj67 testdata.528094 2024-09-13T16:15:49.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:49.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj68 testdata.528094 2024-09-13T16:15:49.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:49.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj69 testdata.528094 2024-09-13T16:15:49.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:49.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj70 testdata.528094 2024-09-13T16:15:50.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:50.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj71 testdata.528094 2024-09-13T16:15:50.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:50.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj72 testdata.528094 2024-09-13T16:15:50.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:50.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj73 testdata.528094 2024-09-13T16:15:50.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:50.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj74 testdata.528094 2024-09-13T16:15:50.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:50.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj75 testdata.528094 2024-09-13T16:15:51.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:51.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj76 testdata.528094 2024-09-13T16:15:51.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:51.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj77 testdata.528094 2024-09-13T16:15:51.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:51.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj78 testdata.528094 2024-09-13T16:15:51.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:51.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj79 testdata.528094 2024-09-13T16:15:51.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:51.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj80 testdata.528094 2024-09-13T16:15:52.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:52.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj81 testdata.528094 2024-09-13T16:15:52.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:52.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj82 testdata.528094 2024-09-13T16:15:52.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:52.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj83 testdata.528094 2024-09-13T16:15:52.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:52.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj84 testdata.528094 2024-09-13T16:15:53.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:53.023 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj85 testdata.528094 2024-09-13T16:15:53.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:53.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj86 testdata.528094 2024-09-13T16:15:53.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:53.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj87 testdata.528094 2024-09-13T16:15:53.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:53.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj88 testdata.528094 2024-09-13T16:15:53.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:53.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj89 testdata.528094 2024-09-13T16:15:54.037 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:54.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj90 testdata.528094 2024-09-13T16:15:54.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:54.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj91 testdata.528094 2024-09-13T16:15:54.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:54.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj92 testdata.528094 2024-09-13T16:15:54.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:54.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj93 testdata.528094 2024-09-13T16:15:54.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:54.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj94 testdata.528094 2024-09-13T16:15:55.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:55.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj95 testdata.528094 2024-09-13T16:15:55.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:55.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj96 testdata.528094 2024-09-13T16:15:55.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:55.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj97 testdata.528094 2024-09-13T16:15:55.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:55.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj98 testdata.528094 2024-09-13T16:15:55.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:55.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj99 testdata.528094 2024-09-13T16:15:56.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:56.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj100 testdata.528094 2024-09-13T16:15:56.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:56.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj101 testdata.528094 2024-09-13T16:15:56.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:56.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj102 testdata.528094 2024-09-13T16:15:57.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:57.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj103 testdata.528094 2024-09-13T16:15:57.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:57.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj104 testdata.528094 2024-09-13T16:15:57.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:57.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj105 testdata.528094 2024-09-13T16:15:57.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:57.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj106 testdata.528094 2024-09-13T16:15:57.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:57.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj107 testdata.528094 2024-09-13T16:15:58.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:58.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj108 testdata.528094 2024-09-13T16:15:58.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:58.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj109 testdata.528094 2024-09-13T16:15:58.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:58.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj110 testdata.528094 2024-09-13T16:15:58.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:58.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj111 testdata.528094 2024-09-13T16:15:59.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:59.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj112 testdata.528094 2024-09-13T16:15:59.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:59.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj113 testdata.528094 2024-09-13T16:15:59.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:59.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj114 testdata.528094 2024-09-13T16:15:59.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:59.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj115 testdata.528094 2024-09-13T16:15:59.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:15:59.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj116 testdata.528094 2024-09-13T16:16:00.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:00.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj117 testdata.528094 2024-09-13T16:16:00.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:00.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj118 testdata.528094 2024-09-13T16:16:00.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:00.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj119 testdata.528094 2024-09-13T16:16:00.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:00.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj120 testdata.528094 2024-09-13T16:16:00.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:00.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj121 testdata.528094 2024-09-13T16:16:01.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:01.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj122 testdata.528094 2024-09-13T16:16:01.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:01.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj123 testdata.528094 2024-09-13T16:16:01.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:01.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj124 testdata.528094 2024-09-13T16:16:01.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:01.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj125 testdata.528094 2024-09-13T16:16:01.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:01.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj126 testdata.528094 2024-09-13T16:16:02.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:02.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj127 testdata.528094 2024-09-13T16:16:02.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:02.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj128 testdata.528094 2024-09-13T16:16:02.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:02.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj129 testdata.528094 2024-09-13T16:16:02.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:02.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj130 testdata.528094 2024-09-13T16:16:03.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:03.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj131 testdata.528094 2024-09-13T16:16:03.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:03.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj132 testdata.528094 2024-09-13T16:16:03.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:03.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj133 testdata.528094 2024-09-13T16:16:03.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:03.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj134 testdata.528094 2024-09-13T16:16:03.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:03.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj135 testdata.528094 2024-09-13T16:16:04.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:04.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj136 testdata.528094 2024-09-13T16:16:04.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:04.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj137 testdata.528094 2024-09-13T16:16:04.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:04.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj138 testdata.528094 2024-09-13T16:16:04.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:04.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj139 testdata.528094 2024-09-13T16:16:04.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:04.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj140 testdata.528094 2024-09-13T16:16:05.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:05.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj141 testdata.528094 2024-09-13T16:16:05.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:05.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj142 testdata.528094 2024-09-13T16:16:05.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:05.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj143 testdata.528094 2024-09-13T16:16:05.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:05.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj144 testdata.528094 2024-09-13T16:16:05.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:05.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj145 testdata.528094 2024-09-13T16:16:06.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:06.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj146 testdata.528094 2024-09-13T16:16:06.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:06.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj147 testdata.528094 2024-09-13T16:16:06.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:06.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj148 testdata.528094 2024-09-13T16:16:06.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:06.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj149 testdata.528094 2024-09-13T16:16:07.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:07.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj150 testdata.528094 2024-09-13T16:16:07.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:07.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj151 testdata.528094 2024-09-13T16:16:07.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:07.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj152 testdata.528094 2024-09-13T16:16:07.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:07.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj153 testdata.528094 2024-09-13T16:16:07.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:07.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj154 testdata.528094 2024-09-13T16:16:08.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:08.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj155 testdata.528094 2024-09-13T16:16:08.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:08.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj156 testdata.528094 2024-09-13T16:16:08.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:08.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj157 testdata.528094 2024-09-13T16:16:08.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:08.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj158 testdata.528094 2024-09-13T16:16:08.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:08.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj159 testdata.528094 2024-09-13T16:16:09.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:09.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj160 testdata.528094 2024-09-13T16:16:09.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:09.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj161 testdata.528094 2024-09-13T16:16:09.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:09.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj162 testdata.528094 2024-09-13T16:16:09.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:09.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj163 testdata.528094 2024-09-13T16:16:10.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:10.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj164 testdata.528094 2024-09-13T16:16:10.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:10.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj165 testdata.528094 2024-09-13T16:16:10.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:10.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj166 testdata.528094 2024-09-13T16:16:10.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:10.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj167 testdata.528094 2024-09-13T16:16:10.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:10.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj168 testdata.528094 2024-09-13T16:16:11.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:11.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj169 testdata.528094 2024-09-13T16:16:11.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:11.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj170 testdata.528094 2024-09-13T16:16:11.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:11.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj171 testdata.528094 2024-09-13T16:16:11.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:11.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj172 testdata.528094 2024-09-13T16:16:11.810 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:11.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj173 testdata.528094 2024-09-13T16:16:12.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:12.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj174 testdata.528094 2024-09-13T16:16:12.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:12.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj175 testdata.528094 2024-09-13T16:16:12.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:12.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj176 testdata.528094 2024-09-13T16:16:12.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:12.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj177 testdata.528094 2024-09-13T16:16:12.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:12.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj178 testdata.528094 2024-09-13T16:16:13.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:13.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj179 testdata.528094 2024-09-13T16:16:13.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:13.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj180 testdata.528094 2024-09-13T16:16:13.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:13.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj181 testdata.528094 2024-09-13T16:16:13.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:13.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj182 testdata.528094 2024-09-13T16:16:13.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:13.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj183 testdata.528094 2024-09-13T16:16:14.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:14.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj184 testdata.528094 2024-09-13T16:16:14.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:14.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj185 testdata.528094 2024-09-13T16:16:14.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:14.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj186 testdata.528094 2024-09-13T16:16:14.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:14.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj187 testdata.528094 2024-09-13T16:16:14.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:14.898 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj188 testdata.528094 2024-09-13T16:16:15.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:15.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj189 testdata.528094 2024-09-13T16:16:15.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:15.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj190 testdata.528094 2024-09-13T16:16:15.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:15.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj191 testdata.528094 2024-09-13T16:16:15.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:15.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj192 testdata.528094 2024-09-13T16:16:15.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:15.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj193 testdata.528094 2024-09-13T16:16:16.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:16.178 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj194 testdata.528094 2024-09-13T16:16:16.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:16.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj195 testdata.528094 2024-09-13T16:16:16.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:16.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj196 testdata.528094 2024-09-13T16:16:16.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:16.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj197 testdata.528094 2024-09-13T16:16:17.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:17.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj198 testdata.528094 2024-09-13T16:16:17.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:17.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj199 testdata.528094 2024-09-13T16:16:17.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:17.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj200 testdata.528094 2024-09-13T16:16:17.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:17.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj201 testdata.528094 2024-09-13T16:16:17.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:17.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj202 testdata.528094 2024-09-13T16:16:18.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:18.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj203 testdata.528094 2024-09-13T16:16:18.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:18.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj204 testdata.528094 2024-09-13T16:16:18.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:18.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj205 testdata.528094 2024-09-13T16:16:18.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:18.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj206 testdata.528094 2024-09-13T16:16:18.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:18.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj207 testdata.528094 2024-09-13T16:16:19.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:19.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj208 testdata.528094 2024-09-13T16:16:19.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:19.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj209 testdata.528094 2024-09-13T16:16:19.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:19.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj210 testdata.528094 2024-09-13T16:16:19.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:19.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj211 testdata.528094 2024-09-13T16:16:20.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:20.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj212 testdata.528094 2024-09-13T16:16:20.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:20.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj213 testdata.528094 2024-09-13T16:16:20.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:20.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj214 testdata.528094 2024-09-13T16:16:20.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:20.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj215 testdata.528094 2024-09-13T16:16:20.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:20.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj216 testdata.528094 2024-09-13T16:16:21.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:21.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj217 testdata.528094 2024-09-13T16:16:21.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:21.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj218 testdata.528094 2024-09-13T16:16:21.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:21.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj219 testdata.528094 2024-09-13T16:16:21.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:21.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj220 testdata.528094 2024-09-13T16:16:21.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:21.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj221 testdata.528094 2024-09-13T16:16:22.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:22.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj222 testdata.528094 2024-09-13T16:16:22.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:22.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj223 testdata.528094 2024-09-13T16:16:22.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:22.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj224 testdata.528094 2024-09-13T16:16:22.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:22.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj225 testdata.528094 2024-09-13T16:16:22.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:22.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj226 testdata.528094 2024-09-13T16:16:23.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:23.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj227 testdata.528094 2024-09-13T16:16:23.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:23.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj228 testdata.528094 2024-09-13T16:16:23.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:23.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj229 testdata.528094 2024-09-13T16:16:23.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:23.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj230 testdata.528094 2024-09-13T16:16:24.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:24.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj231 testdata.528094 2024-09-13T16:16:24.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:24.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj232 testdata.528094 2024-09-13T16:16:24.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:24.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj233 testdata.528094 2024-09-13T16:16:24.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:24.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj234 testdata.528094 2024-09-13T16:16:24.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:24.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj235 testdata.528094 2024-09-13T16:16:25.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:25.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj236 testdata.528094 2024-09-13T16:16:25.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:25.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj237 testdata.528094 2024-09-13T16:16:25.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:25.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj238 testdata.528094 2024-09-13T16:16:25.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:25.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj239 testdata.528094 2024-09-13T16:16:25.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:25.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj240 testdata.528094 2024-09-13T16:16:26.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:26.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj241 testdata.528094 2024-09-13T16:16:26.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:26.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj242 testdata.528094 2024-09-13T16:16:26.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:26.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj243 testdata.528094 2024-09-13T16:16:26.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:26.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj244 testdata.528094 2024-09-13T16:16:26.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:26.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj245 testdata.528094 2024-09-13T16:16:27.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:27.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj246 testdata.528094 2024-09-13T16:16:27.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:27.950 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj247 testdata.528094 2024-09-13T16:16:28.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:28.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj248 testdata.528094 2024-09-13T16:16:28.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:28.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj249 testdata.528094 2024-09-13T16:16:28.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:28.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj250 testdata.528094 2024-09-13T16:16:28.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:28.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj251 testdata.528094 2024-09-13T16:16:28.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:28.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj252 testdata.528094 2024-09-13T16:16:29.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:29.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj253 testdata.528094 2024-09-13T16:16:29.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:29.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj254 testdata.528094 2024-09-13T16:16:29.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:29.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj255 testdata.528094 2024-09-13T16:16:29.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:29.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj256 testdata.528094 2024-09-13T16:16:30.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:30.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj257 testdata.528094 2024-09-13T16:16:30.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:30.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj258 testdata.528094 2024-09-13T16:16:30.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:30.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj259 testdata.528094 2024-09-13T16:16:30.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:30.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj260 testdata.528094 2024-09-13T16:16:30.888 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:30.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj261 testdata.528094 2024-09-13T16:16:31.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:31.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj262 testdata.528094 2024-09-13T16:16:31.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:31.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj263 testdata.528094 2024-09-13T16:16:31.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:31.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj264 testdata.528094 2024-09-13T16:16:31.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:31.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj265 testdata.528094 2024-09-13T16:16:31.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:31.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj266 testdata.528094 2024-09-13T16:16:32.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:32.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj267 testdata.528094 2024-09-13T16:16:32.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:32.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj268 testdata.528094 2024-09-13T16:16:32.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:32.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj269 testdata.528094 2024-09-13T16:16:32.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:32.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj270 testdata.528094 2024-09-13T16:16:32.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:32.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj271 testdata.528094 2024-09-13T16:16:33.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:33.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj272 testdata.528094 2024-09-13T16:16:33.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:33.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj273 testdata.528094 2024-09-13T16:16:33.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:33.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj274 testdata.528094 2024-09-13T16:16:33.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:33.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj275 testdata.528094 2024-09-13T16:16:34.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:34.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj276 testdata.528094 2024-09-13T16:16:34.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:34.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj277 testdata.528094 2024-09-13T16:16:34.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:34.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj278 testdata.528094 2024-09-13T16:16:34.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:34.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj279 testdata.528094 2024-09-13T16:16:34.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:34.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj280 testdata.528094 2024-09-13T16:16:35.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:35.080 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj281 testdata.528094 2024-09-13T16:16:35.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:35.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj282 testdata.528094 2024-09-13T16:16:35.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:35.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj283 testdata.528094 2024-09-13T16:16:35.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:35.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj284 testdata.528094 2024-09-13T16:16:35.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:35.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj285 testdata.528094 2024-09-13T16:16:36.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:36.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj286 testdata.528094 2024-09-13T16:16:36.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:36.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj287 testdata.528094 2024-09-13T16:16:36.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:36.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj288 testdata.528094 2024-09-13T16:16:36.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:36.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj289 testdata.528094 2024-09-13T16:16:37.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:37.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj290 testdata.528094 2024-09-13T16:16:37.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:37.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj291 testdata.528094 2024-09-13T16:16:37.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:37.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj292 testdata.528094 2024-09-13T16:16:37.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:37.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj293 testdata.528094 2024-09-13T16:16:37.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:37.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj294 testdata.528094 2024-09-13T16:16:38.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:38.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj295 testdata.528094 2024-09-13T16:16:38.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:38.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj296 testdata.528094 2024-09-13T16:16:38.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:38.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj297 testdata.528094 2024-09-13T16:16:38.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:38.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj298 testdata.528094 2024-09-13T16:16:38.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:38.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj299 testdata.528094 2024-09-13T16:16:39.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:39.130 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj300 testdata.528094 2024-09-13T16:16:39.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:39.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj301 testdata.528094 2024-09-13T16:16:39.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:39.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj302 testdata.528094 2024-09-13T16:16:39.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:39.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj303 testdata.528094 2024-09-13T16:16:40.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:40.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj304 testdata.528094 2024-09-13T16:16:40.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:40.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj305 testdata.528094 2024-09-13T16:16:40.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:40.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj306 testdata.528094 2024-09-13T16:16:40.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:40.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj307 testdata.528094 2024-09-13T16:16:40.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:40.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj308 testdata.528094 2024-09-13T16:16:41.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:41.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj309 testdata.528094 2024-09-13T16:16:41.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:41.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj310 testdata.528094 2024-09-13T16:16:41.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:41.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj311 testdata.528094 2024-09-13T16:16:41.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:41.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj312 testdata.528094 2024-09-13T16:16:41.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:41.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj313 testdata.528094 2024-09-13T16:16:42.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:42.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj314 testdata.528094 2024-09-13T16:16:42.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:42.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj315 testdata.528094 2024-09-13T16:16:42.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:42.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj316 testdata.528094 2024-09-13T16:16:42.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:42.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj317 testdata.528094 2024-09-13T16:16:42.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:42.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj318 testdata.528094 2024-09-13T16:16:43.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:43.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj319 testdata.528094 2024-09-13T16:16:43.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:43.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj320 testdata.528094 2024-09-13T16:16:43.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:43.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj321 testdata.528094 2024-09-13T16:16:43.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:43.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj322 testdata.528094 2024-09-13T16:16:43.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:43.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj323 testdata.528094 2024-09-13T16:16:44.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:44.192 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj324 testdata.528094 2024-09-13T16:16:44.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:44.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj325 testdata.528094 2024-09-13T16:16:44.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:44.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj326 testdata.528094 2024-09-13T16:16:44.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:44.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj327 testdata.528094 2024-09-13T16:16:45.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:45.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj328 testdata.528094 2024-09-13T16:16:45.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:45.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj329 testdata.528094 2024-09-13T16:16:45.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:45.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj330 testdata.528094 2024-09-13T16:16:45.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:45.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj331 testdata.528094 2024-09-13T16:16:45.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:45.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj332 testdata.528094 2024-09-13T16:16:46.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:46.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj333 testdata.528094 2024-09-13T16:16:46.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:46.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj334 testdata.528094 2024-09-13T16:16:46.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:46.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj335 testdata.528094 2024-09-13T16:16:46.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:46.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj336 testdata.528094 2024-09-13T16:16:46.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:46.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj337 testdata.528094 2024-09-13T16:16:47.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:47.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj338 testdata.528094 2024-09-13T16:16:47.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:47.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj339 testdata.528094 2024-09-13T16:16:47.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:47.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj340 testdata.528094 2024-09-13T16:16:47.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:47.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj341 testdata.528094 2024-09-13T16:16:47.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:47.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj342 testdata.528094 2024-09-13T16:16:48.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:48.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj343 testdata.528094 2024-09-13T16:16:48.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:48.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj344 testdata.528094 2024-09-13T16:16:48.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:48.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj345 testdata.528094 2024-09-13T16:16:48.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:48.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj346 testdata.528094 2024-09-13T16:16:49.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:49.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj347 testdata.528094 2024-09-13T16:16:49.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:49.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj348 testdata.528094 2024-09-13T16:16:49.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:49.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj349 testdata.528094 2024-09-13T16:16:49.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:49.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj350 testdata.528094 2024-09-13T16:16:49.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:49.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj351 testdata.528094 2024-09-13T16:16:50.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:50.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj352 testdata.528094 2024-09-13T16:16:50.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:50.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj353 testdata.528094 2024-09-13T16:16:50.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:50.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj354 testdata.528094 2024-09-13T16:16:50.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:50.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj355 testdata.528094 2024-09-13T16:16:51.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:51.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj356 testdata.528094 2024-09-13T16:16:51.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:51.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj357 testdata.528094 2024-09-13T16:16:51.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:51.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj358 testdata.528094 2024-09-13T16:16:51.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:51.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj359 testdata.528094 2024-09-13T16:16:51.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:51.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj360 testdata.528094 2024-09-13T16:16:52.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:52.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj361 testdata.528094 2024-09-13T16:16:52.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:52.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj362 testdata.528094 2024-09-13T16:16:52.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:52.492 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj363 testdata.528094 2024-09-13T16:16:52.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:52.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj364 testdata.528094 2024-09-13T16:16:52.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:52.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj365 testdata.528094 2024-09-13T16:16:53.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:53.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj366 testdata.528094 2024-09-13T16:16:53.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:53.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj367 testdata.528094 2024-09-13T16:16:53.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:53.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj368 testdata.528094 2024-09-13T16:16:53.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:53.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj369 testdata.528094 2024-09-13T16:16:54.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:54.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj370 testdata.528094 2024-09-13T16:16:54.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:54.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj371 testdata.528094 2024-09-13T16:16:54.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:54.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj372 testdata.528094 2024-09-13T16:16:54.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:54.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj373 testdata.528094 2024-09-13T16:16:54.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:54.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj374 testdata.528094 2024-09-13T16:16:55.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:55.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj375 testdata.528094 2024-09-13T16:16:55.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:55.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj376 testdata.528094 2024-09-13T16:16:55.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:55.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj377 testdata.528094 2024-09-13T16:16:55.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:55.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj378 testdata.528094 2024-09-13T16:16:56.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:56.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj379 testdata.528094 2024-09-13T16:16:56.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:56.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj380 testdata.528094 2024-09-13T16:16:56.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:56.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj381 testdata.528094 2024-09-13T16:16:56.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:56.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj382 testdata.528094 2024-09-13T16:16:56.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:56.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj383 testdata.528094 2024-09-13T16:16:57.144 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:57.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj384 testdata.528094 2024-09-13T16:16:57.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:57.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj385 testdata.528094 2024-09-13T16:16:57.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:57.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj386 testdata.528094 2024-09-13T16:16:57.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:57.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj387 testdata.528094 2024-09-13T16:16:58.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:58.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj388 testdata.528094 2024-09-13T16:16:58.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:58.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj389 testdata.528094 2024-09-13T16:16:58.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:58.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj390 testdata.528094 2024-09-13T16:16:59.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:59.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj391 testdata.528094 2024-09-13T16:16:59.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:59.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj392 testdata.528094 2024-09-13T16:16:59.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:59.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj393 testdata.528094 2024-09-13T16:16:59.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:59.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj394 testdata.528094 2024-09-13T16:16:59.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:16:59.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj395 testdata.528094 2024-09-13T16:17:00.199 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:00.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj396 testdata.528094 2024-09-13T16:17:00.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:00.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj397 testdata.528094 2024-09-13T16:17:00.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:00.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj398 testdata.528094 2024-09-13T16:17:00.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:00.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj399 testdata.528094 2024-09-13T16:17:01.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:01.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj400 testdata.528094 2024-09-13T16:17:01.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:01.329 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj401 testdata.528094 2024-09-13T16:17:01.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:01.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj402 testdata.528094 2024-09-13T16:17:01.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:01.746 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj403 testdata.528094 2024-09-13T16:17:01.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:01.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj404 testdata.528094 2024-09-13T16:17:02.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:02.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj405 testdata.528094 2024-09-13T16:17:02.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:02.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj406 testdata.528094 2024-09-13T16:17:02.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:02.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj407 testdata.528094 2024-09-13T16:17:02.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:02.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj408 testdata.528094 2024-09-13T16:17:03.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:03.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj409 testdata.528094 2024-09-13T16:17:03.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:03.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj410 testdata.528094 2024-09-13T16:17:03.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:03.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj411 testdata.528094 2024-09-13T16:17:03.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:03.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj412 testdata.528094 2024-09-13T16:17:03.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:03.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj413 testdata.528094 2024-09-13T16:17:04.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:04.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj414 testdata.528094 2024-09-13T16:17:04.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:04.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj415 testdata.528094 2024-09-13T16:17:04.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:04.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj416 testdata.528094 2024-09-13T16:17:04.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:04.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj417 testdata.528094 2024-09-13T16:17:04.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:04.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj418 testdata.528094 2024-09-13T16:17:05.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:05.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj419 testdata.528094 2024-09-13T16:17:05.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:05.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj420 testdata.528094 2024-09-13T16:17:05.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:05.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj421 testdata.528094 2024-09-13T16:17:05.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:05.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj422 testdata.528094 2024-09-13T16:17:06.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:06.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj423 testdata.528094 2024-09-13T16:17:06.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:06.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj424 testdata.528094 2024-09-13T16:17:06.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:06.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj425 testdata.528094 2024-09-13T16:17:06.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:06.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj426 testdata.528094 2024-09-13T16:17:06.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:06.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj427 testdata.528094 2024-09-13T16:17:07.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:07.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj428 testdata.528094 2024-09-13T16:17:07.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:07.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj429 testdata.528094 2024-09-13T16:17:07.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:07.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj430 testdata.528094 2024-09-13T16:17:07.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:07.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj431 testdata.528094 2024-09-13T16:17:08.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:08.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj432 testdata.528094 2024-09-13T16:17:08.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:08.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj433 testdata.528094 2024-09-13T16:17:08.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:08.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj434 testdata.528094 2024-09-13T16:17:08.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:08.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj435 testdata.528094 2024-09-13T16:17:08.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:08.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj436 testdata.528094 2024-09-13T16:17:09.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:09.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj437 testdata.528094 2024-09-13T16:17:09.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:09.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj438 testdata.528094 2024-09-13T16:17:09.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:09.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj439 testdata.528094 2024-09-13T16:17:09.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:09.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj440 testdata.528094 2024-09-13T16:17:10.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:10.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj441 testdata.528094 2024-09-13T16:17:10.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:10.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj442 testdata.528094 2024-09-13T16:17:10.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:10.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj443 testdata.528094 2024-09-13T16:17:10.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:10.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj444 testdata.528094 2024-09-13T16:17:10.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:10.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj445 testdata.528094 2024-09-13T16:17:11.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:11.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj446 testdata.528094 2024-09-13T16:17:11.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:11.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj447 testdata.528094 2024-09-13T16:17:11.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:11.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj448 testdata.528094 2024-09-13T16:17:11.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:11.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj449 testdata.528094 2024-09-13T16:17:11.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:11.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj450 testdata.528094 2024-09-13T16:17:12.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:12.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj451 testdata.528094 2024-09-13T16:17:12.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:12.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj452 testdata.528094 2024-09-13T16:17:12.595 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:12.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj453 testdata.528094 2024-09-13T16:17:12.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:12.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj454 testdata.528094 2024-09-13T16:17:13.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:13.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj455 testdata.528094 2024-09-13T16:17:13.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:13.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj456 testdata.528094 2024-09-13T16:17:13.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:13.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj457 testdata.528094 2024-09-13T16:17:13.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:13.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj458 testdata.528094 2024-09-13T16:17:13.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:13.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj459 testdata.528094 2024-09-13T16:17:14.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:14.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj460 testdata.528094 2024-09-13T16:17:14.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:14.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj461 testdata.528094 2024-09-13T16:17:14.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:14.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj462 testdata.528094 2024-09-13T16:17:14.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:14.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj463 testdata.528094 2024-09-13T16:17:15.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:15.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj464 testdata.528094 2024-09-13T16:17:15.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:15.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj465 testdata.528094 2024-09-13T16:17:15.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:15.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj466 testdata.528094 2024-09-13T16:17:15.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:15.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj467 testdata.528094 2024-09-13T16:17:15.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:15.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj468 testdata.528094 2024-09-13T16:17:16.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:16.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj469 testdata.528094 2024-09-13T16:17:16.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:16.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj470 testdata.528094 2024-09-13T16:17:16.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:16.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj471 testdata.528094 2024-09-13T16:17:16.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:16.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj472 testdata.528094 2024-09-13T16:17:16.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:16.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj473 testdata.528094 2024-09-13T16:17:17.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:17.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj474 testdata.528094 2024-09-13T16:17:17.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:17.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj475 testdata.528094 2024-09-13T16:17:17.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:17.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj476 testdata.528094 2024-09-13T16:17:17.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:17.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj477 testdata.528094 2024-09-13T16:17:18.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:18.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj478 testdata.528094 2024-09-13T16:17:18.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:18.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj479 testdata.528094 2024-09-13T16:17:18.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:18.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj480 testdata.528094 2024-09-13T16:17:18.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:18.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj481 testdata.528094 2024-09-13T16:17:18.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:18.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj482 testdata.528094 2024-09-13T16:17:19.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:19.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj483 testdata.528094 2024-09-13T16:17:19.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:19.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj484 testdata.528094 2024-09-13T16:17:19.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:19.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj485 testdata.528094 2024-09-13T16:17:19.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:19.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj486 testdata.528094 2024-09-13T16:17:20.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:20.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj487 testdata.528094 2024-09-13T16:17:20.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:20.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj488 testdata.528094 2024-09-13T16:17:20.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:20.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj489 testdata.528094 2024-09-13T16:17:20.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:20.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj490 testdata.528094 2024-09-13T16:17:20.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:20.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj491 testdata.528094 2024-09-13T16:17:21.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:21.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj492 testdata.528094 2024-09-13T16:17:21.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:21.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj493 testdata.528094 2024-09-13T16:17:21.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:21.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj494 testdata.528094 2024-09-13T16:17:21.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:21.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj495 testdata.528094 2024-09-13T16:17:22.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:22.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj496 testdata.528094 2024-09-13T16:17:22.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:22.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj497 testdata.528094 2024-09-13T16:17:22.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:22.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj498 testdata.528094 2024-09-13T16:17:22.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:22.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj499 testdata.528094 2024-09-13T16:17:22.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:22.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj500 testdata.528094 2024-09-13T16:17:23.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:23.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj501 testdata.528094 2024-09-13T16:17:23.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:23.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj502 testdata.528094 2024-09-13T16:17:23.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:23.645 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj503 testdata.528094 2024-09-13T16:17:23.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:23.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj504 testdata.528094 2024-09-13T16:17:24.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:24.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj505 testdata.528094 2024-09-13T16:17:24.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:24.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj506 testdata.528094 2024-09-13T16:17:24.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:24.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj507 testdata.528094 2024-09-13T16:17:24.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:24.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj508 testdata.528094 2024-09-13T16:17:24.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:24.900 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj509 testdata.528094 2024-09-13T16:17:25.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:25.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj510 testdata.528094 2024-09-13T16:17:25.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:25.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj511 testdata.528094 2024-09-13T16:17:25.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:25.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj512 testdata.528094 2024-09-13T16:17:25.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:25.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj513 testdata.528094 2024-09-13T16:17:26.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:26.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj514 testdata.528094 2024-09-13T16:17:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:26.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj515 testdata.528094 2024-09-13T16:17:26.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:26.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj516 testdata.528094 2024-09-13T16:17:26.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:26.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj517 testdata.528094 2024-09-13T16:17:26.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:26.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj518 testdata.528094 2024-09-13T16:17:27.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:27.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj519 testdata.528094 2024-09-13T16:17:27.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:27.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj520 testdata.528094 2024-09-13T16:17:27.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:27.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj521 testdata.528094 2024-09-13T16:17:27.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:27.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj522 testdata.528094 2024-09-13T16:17:27.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:27.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj523 testdata.528094 2024-09-13T16:17:28.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:28.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj524 testdata.528094 2024-09-13T16:17:28.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:28.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj525 testdata.528094 2024-09-13T16:17:28.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:28.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj526 testdata.528094 2024-09-13T16:17:28.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:28.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj527 testdata.528094 2024-09-13T16:17:29.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:29.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj528 testdata.528094 2024-09-13T16:17:29.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:29.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj529 testdata.528094 2024-09-13T16:17:29.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:29.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj530 testdata.528094 2024-09-13T16:17:30.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:30.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj531 testdata.528094 2024-09-13T16:17:30.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:30.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj532 testdata.528094 2024-09-13T16:17:30.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:30.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj533 testdata.528094 2024-09-13T16:17:30.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:30.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj534 testdata.528094 2024-09-13T16:17:31.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:31.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj535 testdata.528094 2024-09-13T16:17:31.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:31.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj536 testdata.528094 2024-09-13T16:17:31.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:31.540 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj537 testdata.528094 2024-09-13T16:17:31.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:31.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj538 testdata.528094 2024-09-13T16:17:31.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:31.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj539 testdata.528094 2024-09-13T16:17:32.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:32.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj540 testdata.528094 2024-09-13T16:17:32.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:32.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj541 testdata.528094 2024-09-13T16:17:32.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:32.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj542 testdata.528094 2024-09-13T16:17:32.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:32.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj543 testdata.528094 2024-09-13T16:17:33.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:33.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj544 testdata.528094 2024-09-13T16:17:33.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:33.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj545 testdata.528094 2024-09-13T16:17:33.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:33.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj546 testdata.528094 2024-09-13T16:17:33.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:33.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj547 testdata.528094 2024-09-13T16:17:33.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:33.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj548 testdata.528094 2024-09-13T16:17:34.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:34.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj549 testdata.528094 2024-09-13T16:17:34.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:34.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj550 testdata.528094 2024-09-13T16:17:34.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:34.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj551 testdata.528094 2024-09-13T16:17:34.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:34.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj552 testdata.528094 2024-09-13T16:17:35.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:35.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj553 testdata.528094 2024-09-13T16:17:35.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:35.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj554 testdata.528094 2024-09-13T16:17:35.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:35.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj555 testdata.528094 2024-09-13T16:17:35.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:35.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj556 testdata.528094 2024-09-13T16:17:35.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:35.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj557 testdata.528094 2024-09-13T16:17:36.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:36.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj558 testdata.528094 2024-09-13T16:17:36.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:36.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj559 testdata.528094 2024-09-13T16:17:36.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:36.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj560 testdata.528094 2024-09-13T16:17:36.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:36.643 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj561 testdata.528094 2024-09-13T16:17:36.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:36.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj562 testdata.528094 2024-09-13T16:17:37.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:37.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj563 testdata.528094 2024-09-13T16:17:37.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:37.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj564 testdata.528094 2024-09-13T16:17:37.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:37.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj565 testdata.528094 2024-09-13T16:17:37.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:37.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj566 testdata.528094 2024-09-13T16:17:37.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:37.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj567 testdata.528094 2024-09-13T16:17:38.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:38.114 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj568 testdata.528094 2024-09-13T16:17:38.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:38.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj569 testdata.528094 2024-09-13T16:17:38.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:38.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj570 testdata.528094 2024-09-13T16:17:38.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:38.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj571 testdata.528094 2024-09-13T16:17:38.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:38.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj572 testdata.528094 2024-09-13T16:17:39.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:39.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj573 testdata.528094 2024-09-13T16:17:39.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:39.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj574 testdata.528094 2024-09-13T16:17:39.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:39.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj575 testdata.528094 2024-09-13T16:17:39.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:39.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj576 testdata.528094 2024-09-13T16:17:40.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:40.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj577 testdata.528094 2024-09-13T16:17:40.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:40.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj578 testdata.528094 2024-09-13T16:17:40.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:40.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj579 testdata.528094 2024-09-13T16:17:40.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:40.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj580 testdata.528094 2024-09-13T16:17:41.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:41.091 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj581 testdata.528094 2024-09-13T16:17:41.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:41.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj582 testdata.528094 2024-09-13T16:17:41.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:41.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj583 testdata.528094 2024-09-13T16:17:41.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:41.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj584 testdata.528094 2024-09-13T16:17:41.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:41.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj585 testdata.528094 2024-09-13T16:17:42.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:42.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj586 testdata.528094 2024-09-13T16:17:42.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:42.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj587 testdata.528094 2024-09-13T16:17:42.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:42.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj588 testdata.528094 2024-09-13T16:17:42.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:42.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj589 testdata.528094 2024-09-13T16:17:42.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:42.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj590 testdata.528094 2024-09-13T16:17:43.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:43.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj591 testdata.528094 2024-09-13T16:17:43.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:43.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj592 testdata.528094 2024-09-13T16:17:43.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:43.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj593 testdata.528094 2024-09-13T16:17:43.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:43.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj594 testdata.528094 2024-09-13T16:17:44.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:44.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj595 testdata.528094 2024-09-13T16:17:44.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:44.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj596 testdata.528094 2024-09-13T16:17:44.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:44.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj597 testdata.528094 2024-09-13T16:17:44.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:44.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj598 testdata.528094 2024-09-13T16:17:45.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:45.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj599 testdata.528094 2024-09-13T16:17:45.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:45.274 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj600 testdata.528094 2024-09-13T16:17:45.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:45.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj601 testdata.528094 2024-09-13T16:17:45.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:45.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj602 testdata.528094 2024-09-13T16:17:45.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:45.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj603 testdata.528094 2024-09-13T16:17:46.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:46.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj604 testdata.528094 2024-09-13T16:17:46.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:46.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj605 testdata.528094 2024-09-13T16:17:46.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:46.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj606 testdata.528094 2024-09-13T16:17:46.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:46.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj607 testdata.528094 2024-09-13T16:17:47.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:47.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj608 testdata.528094 2024-09-13T16:17:47.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:47.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj609 testdata.528094 2024-09-13T16:17:47.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:47.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj610 testdata.528094 2024-09-13T16:17:47.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:47.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj611 testdata.528094 2024-09-13T16:17:47.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:47.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj612 testdata.528094 2024-09-13T16:17:48.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:48.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj613 testdata.528094 2024-09-13T16:17:48.358 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:48.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj614 testdata.528094 2024-09-13T16:17:48.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:48.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj615 testdata.528094 2024-09-13T16:17:48.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:48.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj616 testdata.528094 2024-09-13T16:17:49.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:49.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj617 testdata.528094 2024-09-13T16:17:49.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:49.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj618 testdata.528094 2024-09-13T16:17:49.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:49.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj619 testdata.528094 2024-09-13T16:17:49.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:49.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj620 testdata.528094 2024-09-13T16:17:49.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:49.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj621 testdata.528094 2024-09-13T16:17:50.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:50.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj622 testdata.528094 2024-09-13T16:17:50.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:50.340 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj623 testdata.528094 2024-09-13T16:17:50.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:50.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj624 testdata.528094 2024-09-13T16:17:50.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:50.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj625 testdata.528094 2024-09-13T16:17:51.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:51.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj626 testdata.528094 2024-09-13T16:17:51.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:51.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj627 testdata.528094 2024-09-13T16:17:51.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:51.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj628 testdata.528094 2024-09-13T16:17:51.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:51.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj629 testdata.528094 2024-09-13T16:17:51.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:51.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj630 testdata.528094 2024-09-13T16:17:52.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:52.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj631 testdata.528094 2024-09-13T16:17:52.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:52.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj632 testdata.528094 2024-09-13T16:17:52.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:52.516 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj633 testdata.528094 2024-09-13T16:17:52.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:52.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj634 testdata.528094 2024-09-13T16:17:52.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:52.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj635 testdata.528094 2024-09-13T16:17:53.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:53.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj636 testdata.528094 2024-09-13T16:17:53.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:53.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj637 testdata.528094 2024-09-13T16:17:53.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:53.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj638 testdata.528094 2024-09-13T16:17:53.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:53.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj639 testdata.528094 2024-09-13T16:17:54.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:54.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj640 testdata.528094 2024-09-13T16:17:54.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:54.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj641 testdata.528094 2024-09-13T16:17:54.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:54.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj642 testdata.528094 2024-09-13T16:17:54.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:54.664 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj643 testdata.528094 2024-09-13T16:17:54.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:54.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj644 testdata.528094 2024-09-13T16:17:55.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:55.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj645 testdata.528094 2024-09-13T16:17:55.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:55.335 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj646 testdata.528094 2024-09-13T16:17:55.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:55.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj647 testdata.528094 2024-09-13T16:17:55.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:55.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj648 testdata.528094 2024-09-13T16:17:55.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:55.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj649 testdata.528094 2024-09-13T16:17:56.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:56.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj650 testdata.528094 2024-09-13T16:17:56.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:56.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj651 testdata.528094 2024-09-13T16:17:56.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:56.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj652 testdata.528094 2024-09-13T16:17:56.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:56.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj653 testdata.528094 2024-09-13T16:17:57.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:57.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj654 testdata.528094 2024-09-13T16:17:57.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:57.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj655 testdata.528094 2024-09-13T16:17:57.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:57.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj656 testdata.528094 2024-09-13T16:17:57.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:57.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj657 testdata.528094 2024-09-13T16:17:58.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:58.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj658 testdata.528094 2024-09-13T16:17:58.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:58.252 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj659 testdata.528094 2024-09-13T16:17:58.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:58.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj660 testdata.528094 2024-09-13T16:17:58.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:58.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj661 testdata.528094 2024-09-13T16:17:58.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:58.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj662 testdata.528094 2024-09-13T16:17:59.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:59.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj663 testdata.528094 2024-09-13T16:17:59.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:59.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj664 testdata.528094 2024-09-13T16:17:59.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:59.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj665 testdata.528094 2024-09-13T16:17:59.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:59.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj666 testdata.528094 2024-09-13T16:17:59.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:17:59.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj667 testdata.528094 2024-09-13T16:18:00.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:00.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj668 testdata.528094 2024-09-13T16:18:00.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:00.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj669 testdata.528094 2024-09-13T16:18:01.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:01.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj670 testdata.528094 2024-09-13T16:18:01.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:01.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj671 testdata.528094 2024-09-13T16:18:01.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:01.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj672 testdata.528094 2024-09-13T16:18:01.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:01.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj673 testdata.528094 2024-09-13T16:18:01.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:01.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj674 testdata.528094 2024-09-13T16:18:02.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:02.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj675 testdata.528094 2024-09-13T16:18:02.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:02.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj676 testdata.528094 2024-09-13T16:18:02.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:02.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj677 testdata.528094 2024-09-13T16:18:02.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:02.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj678 testdata.528094 2024-09-13T16:18:02.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:02.968 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj679 testdata.528094 2024-09-13T16:18:03.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:03.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj680 testdata.528094 2024-09-13T16:18:03.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:03.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj681 testdata.528094 2024-09-13T16:18:03.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:03.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj682 testdata.528094 2024-09-13T16:18:03.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:03.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj683 testdata.528094 2024-09-13T16:18:04.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:04.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj684 testdata.528094 2024-09-13T16:18:04.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:04.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj685 testdata.528094 2024-09-13T16:18:04.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:04.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj686 testdata.528094 2024-09-13T16:18:04.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:04.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj687 testdata.528094 2024-09-13T16:18:04.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:04.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj688 testdata.528094 2024-09-13T16:18:05.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:05.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj689 testdata.528094 2024-09-13T16:18:05.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:05.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj690 testdata.528094 2024-09-13T16:18:05.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:05.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj691 testdata.528094 2024-09-13T16:18:05.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:05.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj692 testdata.528094 2024-09-13T16:18:06.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:06.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj693 testdata.528094 2024-09-13T16:18:06.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:06.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj694 testdata.528094 2024-09-13T16:18:06.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:06.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj695 testdata.528094 2024-09-13T16:18:06.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:06.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj696 testdata.528094 2024-09-13T16:18:06.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:06.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj697 testdata.528094 2024-09-13T16:18:07.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:07.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj698 testdata.528094 2024-09-13T16:18:07.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:07.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj699 testdata.528094 2024-09-13T16:18:07.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:07.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj700 testdata.528094 2024-09-13T16:18:07.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:07.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj701 testdata.528094 2024-09-13T16:18:07.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:07.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj702 testdata.528094 2024-09-13T16:18:08.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:08.145 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj703 testdata.528094 2024-09-13T16:18:08.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:08.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj704 testdata.528094 2024-09-13T16:18:08.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:08.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj705 testdata.528094 2024-09-13T16:18:08.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:08.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj706 testdata.528094 2024-09-13T16:18:09.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:09.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj707 testdata.528094 2024-09-13T16:18:09.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:09.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj708 testdata.528094 2024-09-13T16:18:09.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:09.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj709 testdata.528094 2024-09-13T16:18:09.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:09.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj710 testdata.528094 2024-09-13T16:18:09.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:09.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj711 testdata.528094 2024-09-13T16:18:10.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:10.153 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj712 testdata.528094 2024-09-13T16:18:10.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:10.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj713 testdata.528094 2024-09-13T16:18:10.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:10.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj714 testdata.528094 2024-09-13T16:18:10.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:10.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj715 testdata.528094 2024-09-13T16:18:11.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:11.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj716 testdata.528094 2024-09-13T16:18:11.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:11.342 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj717 testdata.528094 2024-09-13T16:18:11.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:11.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj718 testdata.528094 2024-09-13T16:18:11.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:11.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj719 testdata.528094 2024-09-13T16:18:12.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:12.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj720 testdata.528094 2024-09-13T16:18:12.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:12.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj721 testdata.528094 2024-09-13T16:18:12.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:12.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj722 testdata.528094 2024-09-13T16:18:12.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:12.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj723 testdata.528094 2024-09-13T16:18:12.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:12.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj724 testdata.528094 2024-09-13T16:18:13.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:13.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj725 testdata.528094 2024-09-13T16:18:13.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:13.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj726 testdata.528094 2024-09-13T16:18:13.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:13.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj727 testdata.528094 2024-09-13T16:18:13.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:13.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj728 testdata.528094 2024-09-13T16:18:13.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:13.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj729 testdata.528094 2024-09-13T16:18:14.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:14.160 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj730 testdata.528094 2024-09-13T16:18:14.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:14.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj731 testdata.528094 2024-09-13T16:18:14.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:14.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj732 testdata.528094 2024-09-13T16:18:14.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:14.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj733 testdata.528094 2024-09-13T16:18:15.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:15.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj734 testdata.528094 2024-09-13T16:18:15.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:15.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj735 testdata.528094 2024-09-13T16:18:15.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:15.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj736 testdata.528094 2024-09-13T16:18:15.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:15.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj737 testdata.528094 2024-09-13T16:18:15.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:15.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj738 testdata.528094 2024-09-13T16:18:16.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:16.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj739 testdata.528094 2024-09-13T16:18:16.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:16.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj740 testdata.528094 2024-09-13T16:18:16.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:16.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj741 testdata.528094 2024-09-13T16:18:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj742 testdata.528094 2024-09-13T16:18:17.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:17.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj743 testdata.528094 2024-09-13T16:18:17.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:17.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj744 testdata.528094 2024-09-13T16:18:17.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:17.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj745 testdata.528094 2024-09-13T16:18:17.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:17.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj746 testdata.528094 2024-09-13T16:18:17.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:17.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj747 testdata.528094 2024-09-13T16:18:18.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:18.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj748 testdata.528094 2024-09-13T16:18:18.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:18.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj749 testdata.528094 2024-09-13T16:18:18.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:18.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj750 testdata.528094 2024-09-13T16:18:18.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:18.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj751 testdata.528094 2024-09-13T16:18:19.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:19.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj752 testdata.528094 2024-09-13T16:18:19.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:19.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj753 testdata.528094 2024-09-13T16:18:19.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:19.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj754 testdata.528094 2024-09-13T16:18:19.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:19.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj755 testdata.528094 2024-09-13T16:18:19.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:19.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj756 testdata.528094 2024-09-13T16:18:20.180 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:20.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj757 testdata.528094 2024-09-13T16:18:20.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:20.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj758 testdata.528094 2024-09-13T16:18:20.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:20.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj759 testdata.528094 2024-09-13T16:18:20.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:20.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj760 testdata.528094 2024-09-13T16:18:21.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:21.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj761 testdata.528094 2024-09-13T16:18:21.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:21.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj762 testdata.528094 2024-09-13T16:18:21.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:21.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj763 testdata.528094 2024-09-13T16:18:21.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:21.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj764 testdata.528094 2024-09-13T16:18:21.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:21.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj765 testdata.528094 2024-09-13T16:18:22.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:22.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj766 testdata.528094 2024-09-13T16:18:22.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:22.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj767 testdata.528094 2024-09-13T16:18:22.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:22.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj768 testdata.528094 2024-09-13T16:18:22.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:22.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj769 testdata.528094 2024-09-13T16:18:22.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:22.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj770 testdata.528094 2024-09-13T16:18:23.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:23.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj771 testdata.528094 2024-09-13T16:18:23.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:23.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj772 testdata.528094 2024-09-13T16:18:23.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:23.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj773 testdata.528094 2024-09-13T16:18:23.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:23.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj774 testdata.528094 2024-09-13T16:18:24.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:24.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj775 testdata.528094 2024-09-13T16:18:24.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:24.345 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj776 testdata.528094 2024-09-13T16:18:24.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:24.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj777 testdata.528094 2024-09-13T16:18:24.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:24.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj778 testdata.528094 2024-09-13T16:18:24.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:24.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj779 testdata.528094 2024-09-13T16:18:25.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:25.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj780 testdata.528094 2024-09-13T16:18:25.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:25.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj781 testdata.528094 2024-09-13T16:18:25.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:25.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj782 testdata.528094 2024-09-13T16:18:25.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:25.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj783 testdata.528094 2024-09-13T16:18:26.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:26.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj784 testdata.528094 2024-09-13T16:18:26.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:26.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj785 testdata.528094 2024-09-13T16:18:26.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:26.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj786 testdata.528094 2024-09-13T16:18:26.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:26.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj787 testdata.528094 2024-09-13T16:18:27.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:27.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj788 testdata.528094 2024-09-13T16:18:27.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:27.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj789 testdata.528094 2024-09-13T16:18:27.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:27.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj790 testdata.528094 2024-09-13T16:18:27.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:27.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj791 testdata.528094 2024-09-13T16:18:27.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:27.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj792 testdata.528094 2024-09-13T16:18:28.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:28.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj793 testdata.528094 2024-09-13T16:18:28.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:28.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj794 testdata.528094 2024-09-13T16:18:28.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:28.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj795 testdata.528094 2024-09-13T16:18:28.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:28.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj796 testdata.528094 2024-09-13T16:18:29.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:29.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj797 testdata.528094 2024-09-13T16:18:29.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:29.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj798 testdata.528094 2024-09-13T16:18:29.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:29.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj799 testdata.528094 2024-09-13T16:18:29.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:29.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj800 testdata.528094 2024-09-13T16:18:29.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:29.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj801 testdata.528094 2024-09-13T16:18:30.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:30.141 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj802 testdata.528094 2024-09-13T16:18:30.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:30.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj803 testdata.528094 2024-09-13T16:18:30.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:30.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj804 testdata.528094 2024-09-13T16:18:30.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:30.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj805 testdata.528094 2024-09-13T16:18:31.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:31.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj806 testdata.528094 2024-09-13T16:18:31.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:31.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj807 testdata.528094 2024-09-13T16:18:31.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:31.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj808 testdata.528094 2024-09-13T16:18:31.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:31.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj809 testdata.528094 2024-09-13T16:18:32.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:32.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj810 testdata.528094 2024-09-13T16:18:32.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:32.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj811 testdata.528094 2024-09-13T16:18:32.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:32.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj812 testdata.528094 2024-09-13T16:18:32.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:32.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj813 testdata.528094 2024-09-13T16:18:33.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:33.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj814 testdata.528094 2024-09-13T16:18:33.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:33.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj815 testdata.528094 2024-09-13T16:18:33.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:33.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj816 testdata.528094 2024-09-13T16:18:33.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:33.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj817 testdata.528094 2024-09-13T16:18:33.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:33.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj818 testdata.528094 2024-09-13T16:18:34.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:34.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj819 testdata.528094 2024-09-13T16:18:34.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:34.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj820 testdata.528094 2024-09-13T16:18:34.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:34.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj821 testdata.528094 2024-09-13T16:18:34.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:34.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj822 testdata.528094 2024-09-13T16:18:35.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:35.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj823 testdata.528094 2024-09-13T16:18:35.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:35.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj824 testdata.528094 2024-09-13T16:18:35.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:35.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj825 testdata.528094 2024-09-13T16:18:35.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:35.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj826 testdata.528094 2024-09-13T16:18:35.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:35.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj827 testdata.528094 2024-09-13T16:18:36.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:36.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj828 testdata.528094 2024-09-13T16:18:36.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:36.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj829 testdata.528094 2024-09-13T16:18:36.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:36.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj830 testdata.528094 2024-09-13T16:18:36.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:36.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj831 testdata.528094 2024-09-13T16:18:36.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:36.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj832 testdata.528094 2024-09-13T16:18:37.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:37.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj833 testdata.528094 2024-09-13T16:18:37.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:37.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj834 testdata.528094 2024-09-13T16:18:37.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:37.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj835 testdata.528094 2024-09-13T16:18:37.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:37.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj836 testdata.528094 2024-09-13T16:18:38.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:38.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj837 testdata.528094 2024-09-13T16:18:38.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:38.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj838 testdata.528094 2024-09-13T16:18:38.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:38.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj839 testdata.528094 2024-09-13T16:18:38.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:38.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj840 testdata.528094 2024-09-13T16:18:38.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:38.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj841 testdata.528094 2024-09-13T16:18:39.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:39.197 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj842 testdata.528094 2024-09-13T16:18:39.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:39.415 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj843 testdata.528094 2024-09-13T16:18:39.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:39.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj844 testdata.528094 2024-09-13T16:18:39.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:39.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj845 testdata.528094 2024-09-13T16:18:40.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:40.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj846 testdata.528094 2024-09-13T16:18:40.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:40.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj847 testdata.528094 2024-09-13T16:18:40.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:40.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj848 testdata.528094 2024-09-13T16:18:40.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:40.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj849 testdata.528094 2024-09-13T16:18:40.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:40.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj850 testdata.528094 2024-09-13T16:18:41.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:41.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj851 testdata.528094 2024-09-13T16:18:41.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:41.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj852 testdata.528094 2024-09-13T16:18:41.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:41.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj853 testdata.528094 2024-09-13T16:18:41.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:41.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj854 testdata.528094 2024-09-13T16:18:42.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:42.083 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj855 testdata.528094 2024-09-13T16:18:42.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:42.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj856 testdata.528094 2024-09-13T16:18:42.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:42.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj857 testdata.528094 2024-09-13T16:18:42.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:42.701 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj858 testdata.528094 2024-09-13T16:18:42.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:42.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj859 testdata.528094 2024-09-13T16:18:43.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:43.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj860 testdata.528094 2024-09-13T16:18:43.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:43.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj861 testdata.528094 2024-09-13T16:18:43.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:43.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj862 testdata.528094 2024-09-13T16:18:43.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:43.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj863 testdata.528094 2024-09-13T16:18:44.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:44.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj864 testdata.528094 2024-09-13T16:18:44.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:44.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj865 testdata.528094 2024-09-13T16:18:44.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:44.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj866 testdata.528094 2024-09-13T16:18:44.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:44.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj867 testdata.528094 2024-09-13T16:18:44.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:44.943 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj868 testdata.528094 2024-09-13T16:18:45.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:45.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj869 testdata.528094 2024-09-13T16:18:45.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:45.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj870 testdata.528094 2024-09-13T16:18:45.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:45.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj871 testdata.528094 2024-09-13T16:18:45.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:45.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj872 testdata.528094 2024-09-13T16:18:46.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:46.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj873 testdata.528094 2024-09-13T16:18:46.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:46.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj874 testdata.528094 2024-09-13T16:18:46.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:46.440 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj875 testdata.528094 2024-09-13T16:18:46.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:46.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj876 testdata.528094 2024-09-13T16:18:46.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:46.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj877 testdata.528094 2024-09-13T16:18:47.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:47.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj878 testdata.528094 2024-09-13T16:18:47.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:47.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj879 testdata.528094 2024-09-13T16:18:47.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:47.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj880 testdata.528094 2024-09-13T16:18:47.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:47.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj881 testdata.528094 2024-09-13T16:18:48.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:48.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj882 testdata.528094 2024-09-13T16:18:48.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:48.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj883 testdata.528094 2024-09-13T16:18:48.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:48.430 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj884 testdata.528094 2024-09-13T16:18:48.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:48.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj885 testdata.528094 2024-09-13T16:18:48.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:48.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj886 testdata.528094 2024-09-13T16:18:49.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:49.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj887 testdata.528094 2024-09-13T16:18:49.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:49.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj888 testdata.528094 2024-09-13T16:18:49.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:49.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj889 testdata.528094 2024-09-13T16:18:49.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:49.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj890 testdata.528094 2024-09-13T16:18:49.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:49.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj891 testdata.528094 2024-09-13T16:18:50.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:50.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj892 testdata.528094 2024-09-13T16:18:50.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:50.245 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj893 testdata.528094 2024-09-13T16:18:50.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:50.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj894 testdata.528094 2024-09-13T16:18:50.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:50.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj895 testdata.528094 2024-09-13T16:18:50.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:50.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj896 testdata.528094 2024-09-13T16:18:51.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:51.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj897 testdata.528094 2024-09-13T16:18:51.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:51.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj898 testdata.528094 2024-09-13T16:18:51.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:51.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj899 testdata.528094 2024-09-13T16:18:51.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:51.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj900 testdata.528094 2024-09-13T16:18:52.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:52.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj901 testdata.528094 2024-09-13T16:18:52.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:52.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj902 testdata.528094 2024-09-13T16:18:52.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:52.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj903 testdata.528094 2024-09-13T16:18:52.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:52.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj904 testdata.528094 2024-09-13T16:18:52.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:52.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj905 testdata.528094 2024-09-13T16:18:53.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:53.105 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj906 testdata.528094 2024-09-13T16:18:53.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:53.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj907 testdata.528094 2024-09-13T16:18:53.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:53.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj908 testdata.528094 2024-09-13T16:18:53.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:53.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj909 testdata.528094 2024-09-13T16:18:53.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:53.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj910 testdata.528094 2024-09-13T16:18:54.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:54.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj911 testdata.528094 2024-09-13T16:18:54.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:54.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj912 testdata.528094 2024-09-13T16:18:54.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:54.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj913 testdata.528094 2024-09-13T16:18:54.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:54.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj914 testdata.528094 2024-09-13T16:18:55.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:55.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj915 testdata.528094 2024-09-13T16:18:55.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:55.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj916 testdata.528094 2024-09-13T16:18:55.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:55.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj917 testdata.528094 2024-09-13T16:18:55.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:55.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj918 testdata.528094 2024-09-13T16:18:55.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:55.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj919 testdata.528094 2024-09-13T16:18:56.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:56.183 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj920 testdata.528094 2024-09-13T16:18:56.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:56.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj921 testdata.528094 2024-09-13T16:18:56.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:56.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj922 testdata.528094 2024-09-13T16:18:56.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:56.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj923 testdata.528094 2024-09-13T16:18:57.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:57.086 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj924 testdata.528094 2024-09-13T16:18:57.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:57.287 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj925 testdata.528094 2024-09-13T16:18:57.495 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:57.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj926 testdata.528094 2024-09-13T16:18:57.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:57.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj927 testdata.528094 2024-09-13T16:18:57.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:57.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj928 testdata.528094 2024-09-13T16:18:58.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:58.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj929 testdata.528094 2024-09-13T16:18:58.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:58.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj930 testdata.528094 2024-09-13T16:18:58.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:58.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj931 testdata.528094 2024-09-13T16:18:58.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:58.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj932 testdata.528094 2024-09-13T16:18:59.092 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:59.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj933 testdata.528094 2024-09-13T16:18:59.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:59.311 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj934 testdata.528094 2024-09-13T16:18:59.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:59.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj935 testdata.528094 2024-09-13T16:18:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:59.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj936 testdata.528094 2024-09-13T16:18:59.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:18:59.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj937 testdata.528094 2024-09-13T16:19:00.155 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:00.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj938 testdata.528094 2024-09-13T16:19:00.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:00.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj939 testdata.528094 2024-09-13T16:19:00.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:00.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj940 testdata.528094 2024-09-13T16:19:00.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:00.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj941 testdata.528094 2024-09-13T16:19:00.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:01.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj942 testdata.528094 2024-09-13T16:19:01.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:01.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj943 testdata.528094 2024-09-13T16:19:01.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:01.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj944 testdata.528094 2024-09-13T16:19:01.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:01.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj945 testdata.528094 2024-09-13T16:19:01.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:01.878 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj946 testdata.528094 2024-09-13T16:19:02.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:02.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj947 testdata.528094 2024-09-13T16:19:02.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:02.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj948 testdata.528094 2024-09-13T16:19:02.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:02.976 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj949 testdata.528094 2024-09-13T16:19:03.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:03.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj950 testdata.528094 2024-09-13T16:19:03.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:03.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj951 testdata.528094 2024-09-13T16:19:03.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:03.633 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj952 testdata.528094 2024-09-13T16:19:03.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:03.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj953 testdata.528094 2024-09-13T16:19:04.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:04.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj954 testdata.528094 2024-09-13T16:19:04.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:04.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj955 testdata.528094 2024-09-13T16:19:04.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:04.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj956 testdata.528094 2024-09-13T16:19:04.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:04.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj957 testdata.528094 2024-09-13T16:19:04.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:04.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj958 testdata.528094 2024-09-13T16:19:05.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:05.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj959 testdata.528094 2024-09-13T16:19:05.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:05.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj960 testdata.528094 2024-09-13T16:19:05.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:05.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj961 testdata.528094 2024-09-13T16:19:05.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:05.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj962 testdata.528094 2024-09-13T16:19:05.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:05.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj963 testdata.528094 2024-09-13T16:19:06.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:06.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj964 testdata.528094 2024-09-13T16:19:06.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:06.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj965 testdata.528094 2024-09-13T16:19:06.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:06.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj966 testdata.528094 2024-09-13T16:19:06.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:06.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj967 testdata.528094 2024-09-13T16:19:07.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:07.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj968 testdata.528094 2024-09-13T16:19:07.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:07.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj969 testdata.528094 2024-09-13T16:19:07.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:07.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj970 testdata.528094 2024-09-13T16:19:07.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:07.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj971 testdata.528094 2024-09-13T16:19:07.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:07.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj972 testdata.528094 2024-09-13T16:19:08.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:08.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj973 testdata.528094 2024-09-13T16:19:08.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:08.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj974 testdata.528094 2024-09-13T16:19:08.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:08.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj975 testdata.528094 2024-09-13T16:19:08.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:08.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj976 testdata.528094 2024-09-13T16:19:08.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:08.960 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj977 testdata.528094 2024-09-13T16:19:09.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:09.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj978 testdata.528094 2024-09-13T16:19:09.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:09.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj979 testdata.528094 2024-09-13T16:19:09.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:09.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj980 testdata.528094 2024-09-13T16:19:09.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:09.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj981 testdata.528094 2024-09-13T16:19:10.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:10.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj982 testdata.528094 2024-09-13T16:19:10.364 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:10.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj983 testdata.528094 2024-09-13T16:19:10.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:10.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj984 testdata.528094 2024-09-13T16:19:10.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:10.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj985 testdata.528094 2024-09-13T16:19:10.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:10.983 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj986 testdata.528094 2024-09-13T16:19:11.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:11.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj987 testdata.528094 2024-09-13T16:19:11.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:11.419 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj988 testdata.528094 2024-09-13T16:19:11.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:11.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj989 testdata.528094 2024-09-13T16:19:11.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:11.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj990 testdata.528094 2024-09-13T16:19:12.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:12.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj991 testdata.528094 2024-09-13T16:19:12.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:12.347 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj992 testdata.528094 2024-09-13T16:19:12.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:12.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj993 testdata.528094 2024-09-13T16:19:12.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:12.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj994 testdata.528094 2024-09-13T16:19:13.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:13.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj995 testdata.528094 2024-09-13T16:19:13.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:13.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj996 testdata.528094 2024-09-13T16:19:13.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:13.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj997 testdata.528094 2024-09-13T16:19:13.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:13.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj998 testdata.528094 2024-09-13T16:19:13.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:13.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj999 testdata.528094 2024-09-13T16:19:14.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: for i in `seq 1 $objects` 2024-09-13T16:19:14.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rados -p test put obj1000 testdata.528094 2024-09-13T16:19:14.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:341: _scrub_abort: rm -f testdata.528094 2024-09-13T16:19:14.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:343: _scrub_abort: get_primary test obj1 2024-09-13T16:19:14.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-13T16:19:14.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T16:19:14.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-13T16:19:14.347 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T16:19:14.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:343: _scrub_abort: local primary=1 2024-09-13T16:19:14.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: local pgid=1.0 2024-09-13T16:19:14.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:346: _scrub_abort: ceph tell 1.0 schedule-scrub 2024-09-13T16:19:14.903 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:19:14.903 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T16:19:14.903 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T16:19:14.903 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-06T16:17:34.904618+0000" 2024-09-13T16:19:14.903 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:19:14.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: set -o pipefail 2024-09-13T16:19:14.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:350: _scrub_abort: found=no 2024-09-13T16:19:14.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: seq 0 200 2024-09-13T16:19:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:19:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:353: _scrub_abort: flush_pg_stats 2024-09-13T16:19:14.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:19:14.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:19:15.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:19:15.296 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:19:15.296 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:19:15.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:19:15.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:15.297 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:19:15.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836535 2024-09-13T16:19:15.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836535 2024-09-13T16:19:15.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535' 2024-09-13T16:19:15.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:15.443 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:19:15.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673011 2024-09-13T16:19:15.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673011 2024-09-13T16:19:15.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535 1-42949673011' 2024-09-13T16:19:15.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:15.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:19:15.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509488 2024-09-13T16:19:15.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509488 2024-09-13T16:19:15.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836535 1-42949673011 2-64424509488' 2024-09-13T16:19:15.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:15.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836535 2024-09-13T16:19:15.720 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:15.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:19:15.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836535 2024-09-13T16:19:15.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:15.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836535 2024-09-13T16:19:15.722 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836535 2024-09-13T16:19:15.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-09-13T16:19:15.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:16.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836533 -lt 21474836535 2024-09-13T16:19:16.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:19:17.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:19:17.032 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:17.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836535 -lt 21474836535 2024-09-13T16:19:17.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:17.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:17.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673011 2024-09-13T16:19:17.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:19:17.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673011 2024-09-13T16:19:17.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:17.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673011 2024-09-13T16:19:17.395 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673011 2024-09-13T16:19:17.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2024-09-13T16:19:17.396 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:19:17.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673012 -lt 42949673011 2024-09-13T16:19:17.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:17.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509488 2024-09-13T16:19:17.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:17.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:19:17.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509488 2024-09-13T16:19:17.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:17.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509488 2024-09-13T16:19:17.700 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509488 2024-09-13T16:19:17.700 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509488' 2024-09-13T16:19:17.700 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:19:18.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509488 -lt 64424509488 2024-09-13T16:19:18.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep '^1.0' 2024-09-13T16:19:18.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: ceph pg dump pgs 2024-09-13T16:19:18.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: grep -q scrubbing 2024-09-13T16:19:18.314 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:19:18.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:356: _scrub_abort: found=yes 2024-09-13T16:19:18.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:358: _scrub_abort: break 2024-09-13T16:19:18.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: set +o pipefail 2024-09-13T16:19:18.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:363: _scrub_abort: test yes = no 2024-09-13T16:19:18.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:369: _scrub_abort: ceph osd set noscrub 2024-09-13T16:19:18.772 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is set 2024-09-13T16:19:18.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: '[' scrub = deep-scrub ']' 2024-09-13T16:19:18.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:376: _scrub_abort: set -o pipefail 2024-09-13T16:19:18.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: seq 0 200 2024-09-13T16:19:18.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:19:18.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:19:18.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:19:18.796 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:19:19.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:19:19.123 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:19:19.123 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:19:19.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:19:19.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:19.124 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:19:19.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836536 2024-09-13T16:19:19.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836536 2024-09-13T16:19:19.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536' 2024-09-13T16:19:19.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:19.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:19:19.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673013 2024-09-13T16:19:19.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673013 2024-09-13T16:19:19.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-42949673013' 2024-09-13T16:19:19.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:19.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:19:19.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509490 2024-09-13T16:19:19.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509490 2024-09-13T16:19:19.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-42949673013 2-64424509490' 2024-09-13T16:19:19.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:19.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:19.566 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836536 2024-09-13T16:19:19.567 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:19:19.567 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836536 2024-09-13T16:19:19.567 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:19.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836536 2024-09-13T16:19:19.569 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836536 2024-09-13T16:19:19.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-09-13T16:19:19.569 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:19.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836535 -lt 21474836536 2024-09-13T16:19:19.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:19:20.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:19:20.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:21.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836537 -lt 21474836536 2024-09-13T16:19:21.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:21.206 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673013 2024-09-13T16:19:21.207 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:21.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:19:21.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673013 2024-09-13T16:19:21.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:21.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673013 2024-09-13T16:19:21.210 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673013 2024-09-13T16:19:21.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673013' 2024-09-13T16:19:21.211 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:19:21.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673013 -lt 42949673013 2024-09-13T16:19:21.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:21.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509490 2024-09-13T16:19:21.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:21.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:19:21.516 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509490 2024-09-13T16:19:21.516 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:21.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509490 2024-09-13T16:19:21.517 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509490 2024-09-13T16:19:21.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509490' 2024-09-13T16:19:21.518 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:19:21.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509490 -lt 64424509490 2024-09-13T16:19:21.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:19:21.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:19:21.893 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:19:22.186 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:19:22.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:19:22.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:19:22.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:19:22.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:19:22.202 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:19:22.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:19:22.504 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:19:22.504 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:19:22.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:19:22.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:22.504 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:19:22.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836538 2024-09-13T16:19:22.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836538 2024-09-13T16:19:22.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836538' 2024-09-13T16:19:22.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:22.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:19:22.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673015 2024-09-13T16:19:22.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673015 2024-09-13T16:19:22.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836538 1-42949673015' 2024-09-13T16:19:22.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:22.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:19:22.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509491 2024-09-13T16:19:22.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509491 2024-09-13T16:19:22.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836538 1-42949673015 2-64424509491' 2024-09-13T16:19:22.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:22.930 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836538 2024-09-13T16:19:22.931 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:22.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:19:22.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836538 2024-09-13T16:19:22.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:22.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836538 2024-09-13T16:19:22.933 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836538 2024-09-13T16:19:22.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-09-13T16:19:22.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:23.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836538 -lt 21474836538 2024-09-13T16:19:23.227 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:23.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673015 2024-09-13T16:19:23.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:23.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:19:23.228 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673015 2024-09-13T16:19:23.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:23.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673015 2024-09-13T16:19:23.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673015' 2024-09-13T16:19:23.230 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673015 2024-09-13T16:19:23.230 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:19:23.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673014 -lt 42949673015 2024-09-13T16:19:23.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:19:24.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:19:24.559 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:19:24.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673015 -lt 42949673015 2024-09-13T16:19:24.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:24.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509491 2024-09-13T16:19:24.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:24.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:19:24.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509491 2024-09-13T16:19:24.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:24.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509491 2024-09-13T16:19:24.875 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509491 2024-09-13T16:19:24.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509491' 2024-09-13T16:19:24.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:19:25.195 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509492 -lt 64424509491 2024-09-13T16:19:25.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:19:25.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:19:25.196 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:19:25.480 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:19:25.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:19:25.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:19:25.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:19:25.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:19:25.497 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:19:25.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:19:25.802 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:19:25.802 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:19:25.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:19:25.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:25.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:19:25.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836540 2024-09-13T16:19:25.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836540 2024-09-13T16:19:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540' 2024-09-13T16:19:25.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:25.947 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:19:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673016 2024-09-13T16:19:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673016 2024-09-13T16:19:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-42949673016' 2024-09-13T16:19:26.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:26.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:19:26.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509493 2024-09-13T16:19:26.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509493 2024-09-13T16:19:26.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-42949673016 2-64424509493' 2024-09-13T16:19:26.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:26.224 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836540 2024-09-13T16:19:26.224 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:26.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:19:26.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836540 2024-09-13T16:19:26.225 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:26.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836540 2024-09-13T16:19:26.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-09-13T16:19:26.227 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836540 2024-09-13T16:19:26.227 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:26.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836538 -lt 21474836540 2024-09-13T16:19:26.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:19:27.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:19:27.538 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:27.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836540 -lt 21474836540 2024-09-13T16:19:27.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:27.871 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:27.872 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673016 2024-09-13T16:19:27.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:19:27.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673016 2024-09-13T16:19:27.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:27.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673016 2024-09-13T16:19:27.875 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673016 2024-09-13T16:19:27.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673016' 2024-09-13T16:19:27.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:19:28.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673017 -lt 42949673016 2024-09-13T16:19:28.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:28.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:28.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509493 2024-09-13T16:19:28.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:19:28.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509493 2024-09-13T16:19:28.237 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:28.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509493 2024-09-13T16:19:28.238 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509493 2024-09-13T16:19:28.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509493' 2024-09-13T16:19:28.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:19:28.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509493 -lt 64424509493 2024-09-13T16:19:28.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:19:28.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:19:28.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:19:28.920 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:19:28.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:19:28.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:19:28.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:19:28.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:19:28.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:19:29.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:19:29.304 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:19:29.305 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:19:29.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:19:29.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:29.305 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:19:29.444 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836541 2024-09-13T16:19:29.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836541 2024-09-13T16:19:29.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836541' 2024-09-13T16:19:29.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:29.445 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:19:29.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673018 2024-09-13T16:19:29.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673018 2024-09-13T16:19:29.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836541 1-42949673018' 2024-09-13T16:19:29.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:29.576 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:19:29.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509495 2024-09-13T16:19:29.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509495 2024-09-13T16:19:29.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836541 1-42949673018 2-64424509495' 2024-09-13T16:19:29.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:29.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836541 2024-09-13T16:19:29.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:29.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:19:29.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836541 2024-09-13T16:19:29.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:29.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836541 2024-09-13T16:19:29.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-09-13T16:19:29.712 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836541 2024-09-13T16:19:29.712 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:30.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836540 -lt 21474836541 2024-09-13T16:19:30.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:19:31.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:19:31.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:31.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836542 -lt 21474836541 2024-09-13T16:19:31.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:31.375 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673018 2024-09-13T16:19:31.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:31.377 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:19:31.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673018 2024-09-13T16:19:31.378 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:31.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673018 2024-09-13T16:19:31.379 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673018 2024-09-13T16:19:31.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673018' 2024-09-13T16:19:31.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:19:31.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673018 -lt 42949673018 2024-09-13T16:19:31.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:31.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509495 2024-09-13T16:19:31.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:31.749 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:19:31.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509495 2024-09-13T16:19:31.750 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:31.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509495 2024-09-13T16:19:31.751 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509495 2024-09-13T16:19:31.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509495' 2024-09-13T16:19:31.752 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:19:32.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509495 -lt 64424509495 2024-09-13T16:19:32.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:19:32.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:19:32.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:19:32.442 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:19:32.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:19:32.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:19:32.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:19:32.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:19:32.460 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:19:32.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:19:32.787 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:19:32.787 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:19:32.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:19:32.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:32.788 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:19:32.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836543 2024-09-13T16:19:32.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836543 2024-09-13T16:19:32.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543' 2024-09-13T16:19:32.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:32.982 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:19:33.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673020 2024-09-13T16:19:33.166 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673020 2024-09-13T16:19:33.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-42949673020' 2024-09-13T16:19:33.167 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:33.167 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:19:33.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509497 2024-09-13T16:19:33.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509497 2024-09-13T16:19:33.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-42949673020 2-64424509497' 2024-09-13T16:19:33.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:33.315 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836543 2024-09-13T16:19:33.315 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:33.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:19:33.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836543 2024-09-13T16:19:33.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:33.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836543 2024-09-13T16:19:33.317 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836543 2024-09-13T16:19:33.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-09-13T16:19:33.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:33.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836542 -lt 21474836543 2024-09-13T16:19:33.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:19:34.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:19:34.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:34.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836543 -lt 21474836543 2024-09-13T16:19:34.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:34.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:34.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673020 2024-09-13T16:19:34.967 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:19:34.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673020 2024-09-13T16:19:34.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:34.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673020 2024-09-13T16:19:34.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2024-09-13T16:19:34.970 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673020 2024-09-13T16:19:34.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:19:35.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673020 -lt 42949673020 2024-09-13T16:19:35.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:35.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509497 2024-09-13T16:19:35.268 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:35.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:19:35.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509497 2024-09-13T16:19:35.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:35.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509497 2024-09-13T16:19:35.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509497' 2024-09-13T16:19:35.272 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509497 2024-09-13T16:19:35.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:19:35.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509497 -lt 64424509497 2024-09-13T16:19:35.637 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:19:35.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:19:35.638 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:19:35.958 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:19:35.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:19:35.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:19:35.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:19:35.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:19:35.975 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:19:36.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:19:36.363 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:19:36.363 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:19:36.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:19:36.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:36.363 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:19:36.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836545 2024-09-13T16:19:36.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836545 2024-09-13T16:19:36.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545' 2024-09-13T16:19:36.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:36.502 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:19:36.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673022 2024-09-13T16:19:36.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673022 2024-09-13T16:19:36.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545 1-42949673022' 2024-09-13T16:19:36.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:36.632 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:19:36.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509498 2024-09-13T16:19:36.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509498 2024-09-13T16:19:36.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836545 1-42949673022 2-64424509498' 2024-09-13T16:19:36.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:36.791 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836545 2024-09-13T16:19:36.791 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:36.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:19:36.793 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836545 2024-09-13T16:19:36.793 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:36.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836545 2024-09-13T16:19:36.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836545' 2024-09-13T16:19:36.794 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836545 2024-09-13T16:19:36.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:37.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836545 -lt 21474836545 2024-09-13T16:19:37.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:37.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673022 2024-09-13T16:19:37.110 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:37.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:19:37.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:37.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673022 2024-09-13T16:19:37.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673022 2024-09-13T16:19:37.113 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673022 2024-09-13T16:19:37.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673022' 2024-09-13T16:19:37.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:19:37.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673022 -lt 42949673022 2024-09-13T16:19:37.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:37.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509498 2024-09-13T16:19:37.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:37.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:19:37.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509498 2024-09-13T16:19:37.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:37.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509498 2024-09-13T16:19:37.410 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509498 2024-09-13T16:19:37.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509498' 2024-09-13T16:19:37.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:19:37.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509497 -lt 64424509498 2024-09-13T16:19:37.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:19:38.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:19:38.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:19:39.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509499 -lt 64424509498 2024-09-13T16:19:39.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:19:39.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:19:39.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:19:39.349 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:19:39.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:19:39.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:19:39.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:19:39.365 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:19:39.365 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:19:39.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:19:39.677 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:19:39.677 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:19:39.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:19:39.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:39.677 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:19:39.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836546 2024-09-13T16:19:39.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836546 2024-09-13T16:19:39.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546' 2024-09-13T16:19:39.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:39.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:19:39.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673023 2024-09-13T16:19:39.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673023 2024-09-13T16:19:39.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546 1-42949673023' 2024-09-13T16:19:39.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:39.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:19:40.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509500 2024-09-13T16:19:40.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509500 2024-09-13T16:19:40.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836546 1-42949673023 2-64424509500' 2024-09-13T16:19:40.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:40.123 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:40.123 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836546 2024-09-13T16:19:40.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:19:40.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836546 2024-09-13T16:19:40.125 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:40.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836546 2024-09-13T16:19:40.127 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836546 2024-09-13T16:19:40.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2024-09-13T16:19:40.128 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:40.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836545 -lt 21474836546 2024-09-13T16:19:40.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:19:41.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:19:41.435 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:41.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836547 -lt 21474836546 2024-09-13T16:19:41.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:41.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673023 2024-09-13T16:19:41.757 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:41.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:19:41.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673023 2024-09-13T16:19:41.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:41.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673023 2024-09-13T16:19:41.761 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673023 2024-09-13T16:19:41.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2024-09-13T16:19:41.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:19:42.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673023 -lt 42949673023 2024-09-13T16:19:42.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:42.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509500 2024-09-13T16:19:42.109 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:42.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:19:42.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509500 2024-09-13T16:19:42.111 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:42.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509500 2024-09-13T16:19:42.112 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509500 2024-09-13T16:19:42.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509500' 2024-09-13T16:19:42.112 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:19:42.486 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509500 -lt 64424509500 2024-09-13T16:19:42.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:19:42.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:19:42.487 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:19:42.785 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:19:42.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:19:42.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:19:42.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:19:42.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:19:42.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:19:43.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:19:43.129 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:19:43.129 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:19:43.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:19:43.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:43.129 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:19:43.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836548 2024-09-13T16:19:43.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836548 2024-09-13T16:19:43.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548' 2024-09-13T16:19:43.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:43.303 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:19:43.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673025 2024-09-13T16:19:43.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673025 2024-09-13T16:19:43.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-42949673025' 2024-09-13T16:19:43.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:43.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:19:43.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509502 2024-09-13T16:19:43.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509502 2024-09-13T16:19:43.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-42949673025 2-64424509502' 2024-09-13T16:19:43.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:43.588 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836548 2024-09-13T16:19:43.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:43.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:19:43.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836548 2024-09-13T16:19:43.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:43.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836548 2024-09-13T16:19:43.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-09-13T16:19:43.592 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836548 2024-09-13T16:19:43.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:43.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836547 -lt 21474836548 2024-09-13T16:19:43.883 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:19:44.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:19:44.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:45.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836548 -lt 21474836548 2024-09-13T16:19:45.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:45.215 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673025 2024-09-13T16:19:45.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:45.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:19:45.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673025 2024-09-13T16:19:45.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:45.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673025 2024-09-13T16:19:45.220 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673025 2024-09-13T16:19:45.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673025' 2024-09-13T16:19:45.220 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:19:45.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673025 -lt 42949673025 2024-09-13T16:19:45.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:45.560 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509502 2024-09-13T16:19:45.561 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:45.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:19:45.564 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509502 2024-09-13T16:19:45.564 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:45.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509502 2024-09-13T16:19:45.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509502' 2024-09-13T16:19:45.565 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509502 2024-09-13T16:19:45.565 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:19:45.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509502 -lt 64424509502 2024-09-13T16:19:45.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:19:45.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:19:45.923 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:19:46.225 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:19:46.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:19:46.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:19:46.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:19:46.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:19:46.242 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:19:46.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:19:46.546 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:19:46.546 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:19:46.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:19:46.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:46.547 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:19:46.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836550 2024-09-13T16:19:46.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836550 2024-09-13T16:19:46.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550' 2024-09-13T16:19:46.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:46.731 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:19:46.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673027 2024-09-13T16:19:46.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673027 2024-09-13T16:19:46.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673027' 2024-09-13T16:19:46.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:46.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:19:47.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509503 2024-09-13T16:19:47.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509503 2024-09-13T16:19:47.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-42949673027 2-64424509503' 2024-09-13T16:19:47.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:47.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836550 2024-09-13T16:19:47.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:47.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:19:47.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836550 2024-09-13T16:19:47.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:47.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836550 2024-09-13T16:19:47.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-09-13T16:19:47.005 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836550 2024-09-13T16:19:47.005 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:47.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836549 -lt 21474836550 2024-09-13T16:19:47.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:19:48.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:19:48.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:48.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836549 -lt 21474836550 2024-09-13T16:19:48.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:19:49.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 499 -eq 0 ']' 2024-09-13T16:19:49.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:49.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836550 -lt 21474836550 2024-09-13T16:19:49.991 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:49.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673027 2024-09-13T16:19:49.992 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:49.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:19:49.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673027 2024-09-13T16:19:49.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:49.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673027 2024-09-13T16:19:49.996 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673027 2024-09-13T16:19:49.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673027' 2024-09-13T16:19:49.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:19:50.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673027 -lt 42949673027 2024-09-13T16:19:50.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:50.361 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509503 2024-09-13T16:19:50.361 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:50.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:19:50.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509503 2024-09-13T16:19:50.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:50.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509503 2024-09-13T16:19:50.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2024-09-13T16:19:50.363 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509503 2024-09-13T16:19:50.364 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:19:50.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509504 -lt 64424509503 2024-09-13T16:19:50.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:19:50.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:19:50.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:19:51.044 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:19:51.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:19:51.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:19:51.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:19:51.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:19:51.061 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:19:51.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:19:51.380 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:19:51.380 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:19:51.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:19:51.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:51.381 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:19:51.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836552 2024-09-13T16:19:51.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836552 2024-09-13T16:19:51.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552' 2024-09-13T16:19:51.519 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:51.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:19:51.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673029 2024-09-13T16:19:51.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673029 2024-09-13T16:19:51.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-42949673029' 2024-09-13T16:19:51.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:51.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:19:51.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509505 2024-09-13T16:19:51.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509505 2024-09-13T16:19:51.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-42949673029 2-64424509505' 2024-09-13T16:19:51.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:51.776 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836552 2024-09-13T16:19:51.776 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:51.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:19:51.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836552 2024-09-13T16:19:51.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:51.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836552 2024-09-13T16:19:51.779 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836552 2024-09-13T16:19:51.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836552' 2024-09-13T16:19:51.780 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:52.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836551 -lt 21474836552 2024-09-13T16:19:52.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:19:53.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:19:53.113 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:53.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836552 -lt 21474836552 2024-09-13T16:19:53.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:53.424 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:53.425 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673029 2024-09-13T16:19:53.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:19:53.426 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673029 2024-09-13T16:19:53.426 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:53.428 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673029 2024-09-13T16:19:53.428 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673029 2024-09-13T16:19:53.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673029' 2024-09-13T16:19:53.429 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:19:53.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673029 -lt 42949673029 2024-09-13T16:19:53.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:53.798 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509505 2024-09-13T16:19:53.798 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:53.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:19:53.800 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509505 2024-09-13T16:19:53.800 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:53.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509505 2024-09-13T16:19:53.802 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509505 2024-09-13T16:19:53.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509505' 2024-09-13T16:19:53.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:19:54.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509505 -lt 64424509505 2024-09-13T16:19:54.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:19:54.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:19:54.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:19:54.508 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:19:54.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:19:54.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:19:54.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:19:54.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:19:54.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:19:54.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:19:54.854 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:19:54.854 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:19:54.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:19:54.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:54.855 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:19:55.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836554 2024-09-13T16:19:55.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836554 2024-09-13T16:19:55.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554' 2024-09-13T16:19:55.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:55.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:19:55.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673030 2024-09-13T16:19:55.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673030 2024-09-13T16:19:55.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673030' 2024-09-13T16:19:55.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:55.151 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:19:55.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509507 2024-09-13T16:19:55.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509507 2024-09-13T16:19:55.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-42949673030 2-64424509507' 2024-09-13T16:19:55.291 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:55.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:55.291 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836554 2024-09-13T16:19:55.292 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:19:55.293 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836554 2024-09-13T16:19:55.293 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:55.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836554 2024-09-13T16:19:55.294 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2024-09-13T16:19:55.294 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836554 2024-09-13T16:19:55.294 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:55.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836552 -lt 21474836554 2024-09-13T16:19:55.599 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:19:56.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:19:56.602 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:56.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836554 -lt 21474836554 2024-09-13T16:19:56.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:56.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673030 2024-09-13T16:19:56.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:56.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:19:56.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673030 2024-09-13T16:19:56.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:56.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673030 2024-09-13T16:19:56.925 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673030 2024-09-13T16:19:56.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673030' 2024-09-13T16:19:56.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:19:57.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673031 -lt 42949673030 2024-09-13T16:19:57.253 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:57.254 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:57.254 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509507 2024-09-13T16:19:57.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:19:57.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509507 2024-09-13T16:19:57.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:57.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509507 2024-09-13T16:19:57.257 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509507 2024-09-13T16:19:57.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509507' 2024-09-13T16:19:57.258 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:19:57.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509507 -lt 64424509507 2024-09-13T16:19:57.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:19:57.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:19:57.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:19:57.890 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:19:57.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:19:57.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:19:57.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:19:57.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:19:57.908 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:19:58.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:19:58.217 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:19:58.217 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:19:58.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:19:58.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:58.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:19:58.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836555 2024-09-13T16:19:58.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836555 2024-09-13T16:19:58.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555' 2024-09-13T16:19:58.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:58.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:19:58.550 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673032 2024-09-13T16:19:58.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673032 2024-09-13T16:19:58.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-42949673032' 2024-09-13T16:19:58.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:19:58.551 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:19:58.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509509 2024-09-13T16:19:58.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509509 2024-09-13T16:19:58.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-42949673032 2-64424509509' 2024-09-13T16:19:58.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:58.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836555 2024-09-13T16:19:58.686 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:58.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:19:58.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:58.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836555 2024-09-13T16:19:58.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836555 2024-09-13T16:19:58.689 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836555 2024-09-13T16:19:58.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836555' 2024-09-13T16:19:58.689 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:19:59.004 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836555 -lt 21474836555 2024-09-13T16:19:59.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:59.100 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673032 2024-09-13T16:19:59.100 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:59.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:19:59.100 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673032 2024-09-13T16:19:59.101 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:59.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673032 2024-09-13T16:19:59.101 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673032 2024-09-13T16:19:59.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673032' 2024-09-13T16:19:59.102 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:19:59.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673032 -lt 42949673032 2024-09-13T16:19:59.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:19:59.339 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509509 2024-09-13T16:19:59.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:19:59.341 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:19:59.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509509 2024-09-13T16:19:59.341 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:19:59.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509509 2024-09-13T16:19:59.343 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509509' 2024-09-13T16:19:59.343 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509509 2024-09-13T16:19:59.343 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:19:59.649 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509509 -lt 64424509509 2024-09-13T16:19:59.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:19:59.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:19:59.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:19:59.948 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:19:59.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:19:59.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:19:59.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:19:59.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:19:59.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:00.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:00.324 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:00.324 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:00.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:00.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:00.325 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:00.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836557 2024-09-13T16:20:00.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836557 2024-09-13T16:20:00.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836557' 2024-09-13T16:20:00.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:00.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:00.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673033 2024-09-13T16:20:00.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673033 2024-09-13T16:20:00.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836557 1-42949673033' 2024-09-13T16:20:00.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:00.603 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:00.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509510 2024-09-13T16:20:00.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509510 2024-09-13T16:20:00.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836557 1-42949673033 2-64424509510' 2024-09-13T16:20:00.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:00.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836557 2024-09-13T16:20:00.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:00.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:00.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836557 2024-09-13T16:20:00.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:00.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836557 2024-09-13T16:20:00.744 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836557 2024-09-13T16:20:00.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836557' 2024-09-13T16:20:00.745 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:01.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836557 -lt 21474836557 2024-09-13T16:20:01.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:01.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673033 2024-09-13T16:20:01.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:01.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:01.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673033 2024-09-13T16:20:01.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:01.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673033 2024-09-13T16:20:01.048 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673033 2024-09-13T16:20:01.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673033' 2024-09-13T16:20:01.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:01.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673033 -lt 42949673033 2024-09-13T16:20:01.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:01.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509510 2024-09-13T16:20:01.402 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:01.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:01.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509510 2024-09-13T16:20:01.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:01.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509510 2024-09-13T16:20:01.406 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509510 2024-09-13T16:20:01.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-09-13T16:20:01.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:01.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509509 -lt 64424509510 2024-09-13T16:20:01.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:20:02.756 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:20:02.757 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:03.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509510 -lt 64424509510 2024-09-13T16:20:03.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:03.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:03.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:03.371 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:03.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:03.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:03.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:03.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:03.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:03.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:03.686 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:03.686 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:03.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:03.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:03.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:03.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836558 2024-09-13T16:20:03.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836558 2024-09-13T16:20:03.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558' 2024-09-13T16:20:03.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:03.830 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:03.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673035 2024-09-13T16:20:03.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673035 2024-09-13T16:20:03.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673035' 2024-09-13T16:20:03.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:03.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:04.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509512 2024-09-13T16:20:04.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509512 2024-09-13T16:20:04.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836558 1-42949673035 2-64424509512' 2024-09-13T16:20:04.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:04.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836558 2024-09-13T16:20:04.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:04.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:04.141 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836558 2024-09-13T16:20:04.141 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:04.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836558 2024-09-13T16:20:04.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2024-09-13T16:20:04.142 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836558 2024-09-13T16:20:04.143 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:04.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836557 -lt 21474836558 2024-09-13T16:20:04.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:20:05.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:20:05.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:05.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836558 -lt 21474836558 2024-09-13T16:20:05.747 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:05.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673035 2024-09-13T16:20:05.748 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:05.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:05.751 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:05.751 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673035 2024-09-13T16:20:05.752 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673035 2024-09-13T16:20:05.753 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673035 2024-09-13T16:20:05.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-09-13T16:20:05.753 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:06.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673035 -lt 42949673035 2024-09-13T16:20:06.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:06.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509512 2024-09-13T16:20:06.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:06.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:06.051 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509512 2024-09-13T16:20:06.051 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:06.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509512 2024-09-13T16:20:06.053 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509512 2024-09-13T16:20:06.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2024-09-13T16:20:06.053 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:06.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509512 -lt 64424509512 2024-09-13T16:20:06.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:06.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:06.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:06.721 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:06.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:06.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:06.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:06.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:06.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:07.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:07.065 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:07.065 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:07.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:07.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:07.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:07.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836560 2024-09-13T16:20:07.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836560 2024-09-13T16:20:07.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560' 2024-09-13T16:20:07.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:07.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:07.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673037 2024-09-13T16:20:07.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673037 2024-09-13T16:20:07.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673037' 2024-09-13T16:20:07.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:07.350 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:07.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509513 2024-09-13T16:20:07.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509513 2024-09-13T16:20:07.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673037 2-64424509513' 2024-09-13T16:20:07.479 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:07.480 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836560 2024-09-13T16:20:07.480 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:07.480 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:07.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836560 2024-09-13T16:20:07.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:07.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836560 2024-09-13T16:20:07.482 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836560 2024-09-13T16:20:07.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836560' 2024-09-13T16:20:07.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:07.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836559 -lt 21474836560 2024-09-13T16:20:07.772 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:20:08.774 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:20:08.775 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:09.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836560 -lt 21474836560 2024-09-13T16:20:09.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:09.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673037 2024-09-13T16:20:09.119 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:09.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:09.120 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673037 2024-09-13T16:20:09.120 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:09.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673037 2024-09-13T16:20:09.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673037' 2024-09-13T16:20:09.122 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673037 2024-09-13T16:20:09.122 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:09.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673037 -lt 42949673037 2024-09-13T16:20:09.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:09.449 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:09.449 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509513 2024-09-13T16:20:09.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:09.449 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509513 2024-09-13T16:20:09.449 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:09.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509513 2024-09-13T16:20:09.450 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509513 2024-09-13T16:20:09.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509513' 2024-09-13T16:20:09.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:09.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509514 -lt 64424509513 2024-09-13T16:20:09.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:09.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:09.758 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:10.054 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:10.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:10.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:10.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:10.071 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:10.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:10.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:10.369 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:10.369 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:10.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:10.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:10.370 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:10.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836562 2024-09-13T16:20:10.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836562 2024-09-13T16:20:10.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562' 2024-09-13T16:20:10.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:10.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:10.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673038 2024-09-13T16:20:10.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673038 2024-09-13T16:20:10.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673038' 2024-09-13T16:20:10.648 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:10.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:10.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509515 2024-09-13T16:20:10.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509515 2024-09-13T16:20:10.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836562 1-42949673038 2-64424509515' 2024-09-13T16:20:10.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:10.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836562 2024-09-13T16:20:10.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:10.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:10.816 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836562 2024-09-13T16:20:10.816 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:10.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836562 2024-09-13T16:20:10.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836562' 2024-09-13T16:20:10.818 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836562 2024-09-13T16:20:10.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:11.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836562 -lt 21474836562 2024-09-13T16:20:11.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:11.120 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673038 2024-09-13T16:20:11.121 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:11.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:11.123 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673038 2024-09-13T16:20:11.123 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:11.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673038 2024-09-13T16:20:11.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-09-13T16:20:11.125 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673038 2024-09-13T16:20:11.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:11.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673038 -lt 42949673038 2024-09-13T16:20:11.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:11.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509515 2024-09-13T16:20:11.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:11.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:11.430 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509515 2024-09-13T16:20:11.430 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:11.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509515 2024-09-13T16:20:11.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509515' 2024-09-13T16:20:11.431 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509515 2024-09-13T16:20:11.431 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:11.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509514 -lt 64424509515 2024-09-13T16:20:11.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:20:12.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:20:12.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:13.113 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509515 -lt 64424509515 2024-09-13T16:20:13.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:13.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:13.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:13.406 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:13.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:13.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:13.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:13.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:13.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:13.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:13.730 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:13.731 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:13.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:13.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:13.731 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:13.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836563 2024-09-13T16:20:13.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836563 2024-09-13T16:20:13.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563' 2024-09-13T16:20:13.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:13.874 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:14.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673040 2024-09-13T16:20:14.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673040 2024-09-13T16:20:14.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673040' 2024-09-13T16:20:14.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:14.000 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:14.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509517 2024-09-13T16:20:14.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509517 2024-09-13T16:20:14.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836563 1-42949673040 2-64424509517' 2024-09-13T16:20:14.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:14.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836563 2024-09-13T16:20:14.133 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:14.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:14.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836563 2024-09-13T16:20:14.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:14.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836563 2024-09-13T16:20:14.135 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836563 2024-09-13T16:20:14.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836563' 2024-09-13T16:20:14.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:14.425 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836562 -lt 21474836563 2024-09-13T16:20:14.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:20:15.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:20:15.428 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:15.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836563 -lt 21474836563 2024-09-13T16:20:15.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:15.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673040 2024-09-13T16:20:15.741 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:15.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:15.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673040 2024-09-13T16:20:15.743 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:15.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673040 2024-09-13T16:20:15.744 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673040 2024-09-13T16:20:15.745 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-09-13T16:20:15.745 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:16.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673040 -lt 42949673040 2024-09-13T16:20:16.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:16.051 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509517 2024-09-13T16:20:16.051 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:16.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:16.053 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:16.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509517 2024-09-13T16:20:16.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509517 2024-09-13T16:20:16.055 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509517 2024-09-13T16:20:16.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509517' 2024-09-13T16:20:16.055 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:16.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509517 -lt 64424509517 2024-09-13T16:20:16.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:16.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:16.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:16.637 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:16.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:16.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:16.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:16.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:16.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:16.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:16.981 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:16.981 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:16.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:16.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:16.981 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:17.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836565 2024-09-13T16:20:17.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836565 2024-09-13T16:20:17.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565' 2024-09-13T16:20:17.119 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:17.119 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:17.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673042 2024-09-13T16:20:17.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673042 2024-09-13T16:20:17.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673042' 2024-09-13T16:20:17.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:17.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:17.371 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509518 2024-09-13T16:20:17.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509518 2024-09-13T16:20:17.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673042 2-64424509518' 2024-09-13T16:20:17.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:17.372 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836565 2024-09-13T16:20:17.372 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:17.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:17.373 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836565 2024-09-13T16:20:17.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:17.374 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836565 2024-09-13T16:20:17.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836565 2024-09-13T16:20:17.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-09-13T16:20:17.375 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:17.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836564 -lt 21474836565 2024-09-13T16:20:17.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:20:18.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:20:18.678 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:18.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836565 -lt 21474836565 2024-09-13T16:20:18.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:18.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673042 2024-09-13T16:20:18.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:18.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:18.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673042 2024-09-13T16:20:18.992 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:18.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673042 2024-09-13T16:20:18.994 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673042 2024-09-13T16:20:18.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673042' 2024-09-13T16:20:18.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:19.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673042 -lt 42949673042 2024-09-13T16:20:19.300 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:19.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509518 2024-09-13T16:20:19.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:19.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:19.303 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:19.303 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509518 2024-09-13T16:20:19.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509518 2024-09-13T16:20:19.304 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509518 2024-09-13T16:20:19.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2024-09-13T16:20:19.305 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:19.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509519 -lt 64424509518 2024-09-13T16:20:19.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:19.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:19.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:19.920 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:19.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:19.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:19.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:19.936 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:19.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:20.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:20.257 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:20.257 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:20.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:20.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:20.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:20.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836567 2024-09-13T16:20:20.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836567 2024-09-13T16:20:20.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567' 2024-09-13T16:20:20.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:20.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:20.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673043 2024-09-13T16:20:20.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673043 2024-09-13T16:20:20.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673043' 2024-09-13T16:20:20.547 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:20.547 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:20.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509520 2024-09-13T16:20:20.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509520 2024-09-13T16:20:20.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509520' 2024-09-13T16:20:20.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:20.677 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836567 2024-09-13T16:20:20.677 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:20.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:20.678 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836567 2024-09-13T16:20:20.678 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:20.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836567 2024-09-13T16:20:20.679 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836567 2024-09-13T16:20:20.679 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2024-09-13T16:20:20.680 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:21.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836567 -lt 21474836567 2024-09-13T16:20:21.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:21.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673043 2024-09-13T16:20:21.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:21.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:21.011 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673043 2024-09-13T16:20:21.011 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:21.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673043 2024-09-13T16:20:21.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-09-13T16:20:21.012 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673043 2024-09-13T16:20:21.012 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:21.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673043 -lt 42949673043 2024-09-13T16:20:21.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:21.314 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509520 2024-09-13T16:20:21.314 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:21.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:21.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509520 2024-09-13T16:20:21.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:21.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509520 2024-09-13T16:20:21.321 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509520 2024-09-13T16:20:21.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-09-13T16:20:21.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:21.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509520 -lt 64424509520 2024-09-13T16:20:21.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:21.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:21.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:21.914 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:21.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:21.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:21.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:21.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:21.933 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:22.254 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:22.255 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:22.255 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:22.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:22.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:22.255 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:22.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836568 2024-09-13T16:20:22.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836568 2024-09-13T16:20:22.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568' 2024-09-13T16:20:22.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:22.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:22.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673045 2024-09-13T16:20:22.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673045 2024-09-13T16:20:22.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568 1-42949673045' 2024-09-13T16:20:22.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:22.533 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:22.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509521 2024-09-13T16:20:22.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509521 2024-09-13T16:20:22.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836568 1-42949673045 2-64424509521' 2024-09-13T16:20:22.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:22.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836568 2024-09-13T16:20:22.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:22.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:22.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836568 2024-09-13T16:20:22.661 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:22.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836568 2024-09-13T16:20:22.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2024-09-13T16:20:22.662 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836568 2024-09-13T16:20:22.662 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:22.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836568 -lt 21474836568 2024-09-13T16:20:22.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:22.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673045 2024-09-13T16:20:22.983 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:22.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:22.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673045 2024-09-13T16:20:22.986 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:22.987 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673045 2024-09-13T16:20:22.987 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673045 2024-09-13T16:20:22.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-09-13T16:20:22.988 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:23.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673045 -lt 42949673045 2024-09-13T16:20:23.298 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:23.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509521 2024-09-13T16:20:23.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:23.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:23.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509521 2024-09-13T16:20:23.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:23.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509521 2024-09-13T16:20:23.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-09-13T16:20:23.302 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509521 2024-09-13T16:20:23.302 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:23.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509521 -lt 64424509521 2024-09-13T16:20:23.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:23.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:23.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:23.886 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:23.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:23.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:23.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:23.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:23.904 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:24.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:24.207 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:24.207 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:24.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:24.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:24.207 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:24.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836569 2024-09-13T16:20:24.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836569 2024-09-13T16:20:24.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569' 2024-09-13T16:20:24.344 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:24.345 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:24.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673046 2024-09-13T16:20:24.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673046 2024-09-13T16:20:24.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673046' 2024-09-13T16:20:24.482 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:24.482 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:24.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509523 2024-09-13T16:20:24.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509523 2024-09-13T16:20:24.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673046 2-64424509523' 2024-09-13T16:20:24.616 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:24.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836569 2024-09-13T16:20:24.616 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:24.617 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:24.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836569 2024-09-13T16:20:24.617 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:24.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836569 2024-09-13T16:20:24.618 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836569 2024-09-13T16:20:24.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-09-13T16:20:24.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:24.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836569 -lt 21474836569 2024-09-13T16:20:24.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:24.907 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673046 2024-09-13T16:20:24.908 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:24.909 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:24.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673046 2024-09-13T16:20:24.910 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:24.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673046 2024-09-13T16:20:24.912 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673046 2024-09-13T16:20:24.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-09-13T16:20:24.912 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:25.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673046 -lt 42949673046 2024-09-13T16:20:25.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:25.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509523 2024-09-13T16:20:25.235 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:25.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:25.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509523 2024-09-13T16:20:25.238 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:25.239 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509523 2024-09-13T16:20:25.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509523 2024-09-13T16:20:25.239 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509523' 2024-09-13T16:20:25.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:25.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509523 -lt 64424509523 2024-09-13T16:20:25.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:25.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:25.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:25.876 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:25.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:25.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:25.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:25.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:25.893 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:26.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:26.203 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:26.204 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:26.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:26.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:26.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:26.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836571 2024-09-13T16:20:26.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836571 2024-09-13T16:20:26.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571' 2024-09-13T16:20:26.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:26.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:26.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673047 2024-09-13T16:20:26.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673047 2024-09-13T16:20:26.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673047' 2024-09-13T16:20:26.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:26.485 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:26.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509524 2024-09-13T16:20:26.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509524 2024-09-13T16:20:26.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836571 1-42949673047 2-64424509524' 2024-09-13T16:20:26.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:26.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836571 2024-09-13T16:20:26.622 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:26.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:26.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836571 2024-09-13T16:20:26.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:26.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836571 2024-09-13T16:20:26.624 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836571 2024-09-13T16:20:26.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-09-13T16:20:26.625 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:26.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836571 -lt 21474836571 2024-09-13T16:20:26.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:26.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:26.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673047 2024-09-13T16:20:26.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:26.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673047 2024-09-13T16:20:26.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:26.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673047 2024-09-13T16:20:26.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673047' 2024-09-13T16:20:26.938 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673047 2024-09-13T16:20:26.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:27.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673048 -lt 42949673047 2024-09-13T16:20:27.272 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:27.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509524 2024-09-13T16:20:27.273 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:27.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:27.276 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509524 2024-09-13T16:20:27.276 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:27.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509524 2024-09-13T16:20:27.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-09-13T16:20:27.278 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509524 2024-09-13T16:20:27.278 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:27.660 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509524 -lt 64424509524 2024-09-13T16:20:27.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:27.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:27.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:27.968 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:27.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:27.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:27.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:27.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:27.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:28.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:28.321 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:28.321 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:28.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:28.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:28.322 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:28.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836572 2024-09-13T16:20:28.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836572 2024-09-13T16:20:28.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572' 2024-09-13T16:20:28.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:28.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:28.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673049 2024-09-13T16:20:28.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673049 2024-09-13T16:20:28.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673049' 2024-09-13T16:20:28.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:28.601 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:28.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509526 2024-09-13T16:20:28.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509526 2024-09-13T16:20:28.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836572 1-42949673049 2-64424509526' 2024-09-13T16:20:28.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:28.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836572 2024-09-13T16:20:28.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:28.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:28.734 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836572 2024-09-13T16:20:28.734 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:28.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836572 2024-09-13T16:20:28.735 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836572 2024-09-13T16:20:28.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-09-13T16:20:28.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:29.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836572 -lt 21474836572 2024-09-13T16:20:29.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:29.053 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673049 2024-09-13T16:20:29.053 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:29.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:29.055 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673049 2024-09-13T16:20:29.055 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:29.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673049 2024-09-13T16:20:29.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-09-13T16:20:29.057 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673049 2024-09-13T16:20:29.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:29.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673049 -lt 42949673049 2024-09-13T16:20:29.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:29.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509526 2024-09-13T16:20:29.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:29.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:29.396 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:29.396 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509526 2024-09-13T16:20:29.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509526 2024-09-13T16:20:29.397 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509526 2024-09-13T16:20:29.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-09-13T16:20:29.398 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:29.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509525 -lt 64424509526 2024-09-13T16:20:29.767 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:20:30.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:20:30.769 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:31.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509526 -lt 64424509526 2024-09-13T16:20:31.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:31.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:31.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:31.419 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:31.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:31.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:31.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:31.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:31.436 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:31.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:31.741 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:31.741 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:31.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:31.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:31.742 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:31.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836574 2024-09-13T16:20:31.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836574 2024-09-13T16:20:31.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574' 2024-09-13T16:20:31.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:31.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:32.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673051 2024-09-13T16:20:32.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673051 2024-09-13T16:20:32.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673051' 2024-09-13T16:20:32.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:32.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:32.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509527 2024-09-13T16:20:32.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509527 2024-09-13T16:20:32.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673051 2-64424509527' 2024-09-13T16:20:32.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:32.150 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836574 2024-09-13T16:20:32.150 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:32.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:32.151 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836574 2024-09-13T16:20:32.151 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:32.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836574 2024-09-13T16:20:32.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2024-09-13T16:20:32.153 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836574 2024-09-13T16:20:32.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:32.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836573 -lt 21474836574 2024-09-13T16:20:32.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:20:33.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:20:33.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:33.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836574 -lt 21474836574 2024-09-13T16:20:33.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:33.778 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673051 2024-09-13T16:20:33.779 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:33.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:33.781 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673051 2024-09-13T16:20:33.781 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:33.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673051 2024-09-13T16:20:33.783 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673051 2024-09-13T16:20:33.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-09-13T16:20:33.783 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:34.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673051 -lt 42949673051 2024-09-13T16:20:34.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:34.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:34.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509527 2024-09-13T16:20:34.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:34.096 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509527 2024-09-13T16:20:34.096 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:34.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509527 2024-09-13T16:20:34.097 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509527 2024-09-13T16:20:34.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2024-09-13T16:20:34.098 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:34.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509527 -lt 64424509527 2024-09-13T16:20:34.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:34.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:34.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:34.708 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:34.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:34.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:34.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:34.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:34.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:35.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:35.048 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:35.048 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:35.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:35.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:35.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:35.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836576 2024-09-13T16:20:35.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836576 2024-09-13T16:20:35.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576' 2024-09-13T16:20:35.194 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:35.194 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:35.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673052 2024-09-13T16:20:35.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673052 2024-09-13T16:20:35.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673052' 2024-09-13T16:20:35.324 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:35.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:35.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509529 2024-09-13T16:20:35.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509529 2024-09-13T16:20:35.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836576 1-42949673052 2-64424509529' 2024-09-13T16:20:35.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:35.462 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:35.462 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836576 2024-09-13T16:20:35.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:35.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836576 2024-09-13T16:20:35.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:35.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836576 2024-09-13T16:20:35.465 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836576 2024-09-13T16:20:35.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836576' 2024-09-13T16:20:35.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:35.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836574 -lt 21474836576 2024-09-13T16:20:35.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:20:36.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:20:36.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:37.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836576 -lt 21474836576 2024-09-13T16:20:37.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:37.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673052 2024-09-13T16:20:37.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:37.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:37.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673052 2024-09-13T16:20:37.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:37.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673052 2024-09-13T16:20:37.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-09-13T16:20:37.073 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673052 2024-09-13T16:20:37.073 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:37.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673053 -lt 42949673052 2024-09-13T16:20:37.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:37.388 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509529 2024-09-13T16:20:37.389 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:37.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:37.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509529 2024-09-13T16:20:37.391 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:37.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509529 2024-09-13T16:20:37.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-09-13T16:20:37.393 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509529 2024-09-13T16:20:37.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:37.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509529 -lt 64424509529 2024-09-13T16:20:37.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:37.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:37.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:37.980 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:37.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:37.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:37.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:37.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:37.996 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:38.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:38.308 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:38.308 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:38.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:38.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:38.309 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:38.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836577 2024-09-13T16:20:38.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836577 2024-09-13T16:20:38.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577' 2024-09-13T16:20:38.453 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:38.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:38.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673054 2024-09-13T16:20:38.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673054 2024-09-13T16:20:38.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673054' 2024-09-13T16:20:38.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:38.591 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:38.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509531 2024-09-13T16:20:38.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509531 2024-09-13T16:20:38.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836577 1-42949673054 2-64424509531' 2024-09-13T16:20:38.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:38.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836577 2024-09-13T16:20:38.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:38.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:38.732 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836577 2024-09-13T16:20:38.732 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:38.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836577 2024-09-13T16:20:38.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-09-13T16:20:38.733 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836577 2024-09-13T16:20:38.733 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:39.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836577 -lt 21474836577 2024-09-13T16:20:39.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:39.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673054 2024-09-13T16:20:39.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:39.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:39.027 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673054 2024-09-13T16:20:39.027 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:39.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673054 2024-09-13T16:20:39.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-09-13T16:20:39.029 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673054 2024-09-13T16:20:39.029 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:39.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673054 -lt 42949673054 2024-09-13T16:20:39.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:39.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509531 2024-09-13T16:20:39.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:39.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:39.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509531 2024-09-13T16:20:39.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:39.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509531 2024-09-13T16:20:39.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2024-09-13T16:20:39.354 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509531 2024-09-13T16:20:39.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:39.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509530 -lt 64424509531 2024-09-13T16:20:39.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:20:40.674 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:20:40.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:40.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509531 -lt 64424509531 2024-09-13T16:20:40.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:40.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:40.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:41.283 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:41.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:41.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:41.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:41.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:41.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:41.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:41.593 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:41.593 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:41.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:41.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:41.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:41.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836579 2024-09-13T16:20:41.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836579 2024-09-13T16:20:41.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579' 2024-09-13T16:20:41.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:41.728 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:41.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673056 2024-09-13T16:20:41.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673056 2024-09-13T16:20:41.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673056' 2024-09-13T16:20:41.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:41.866 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:42.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509532 2024-09-13T16:20:42.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509532 2024-09-13T16:20:42.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673056 2-64424509532' 2024-09-13T16:20:42.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:42.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836579 2024-09-13T16:20:42.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:42.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:42.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836579 2024-09-13T16:20:42.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:42.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836579 2024-09-13T16:20:42.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-09-13T16:20:42.005 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836579 2024-09-13T16:20:42.005 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:42.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836578 -lt 21474836579 2024-09-13T16:20:42.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:20:43.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:20:43.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:43.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836579 -lt 21474836579 2024-09-13T16:20:43.650 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:43.651 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673056 2024-09-13T16:20:43.651 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:43.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:43.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673056 2024-09-13T16:20:43.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:43.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673056 2024-09-13T16:20:43.654 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673056 2024-09-13T16:20:43.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-09-13T16:20:43.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:44.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673056 -lt 42949673056 2024-09-13T16:20:44.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:44.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509532 2024-09-13T16:20:44.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:44.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:44.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509532 2024-09-13T16:20:44.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:44.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509532 2024-09-13T16:20:44.014 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509532 2024-09-13T16:20:44.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-09-13T16:20:44.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:44.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509532 -lt 64424509532 2024-09-13T16:20:44.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:44.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:44.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:44.616 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:44.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:44.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:44.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:44.632 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:44.632 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:45.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:45.001 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:45.001 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:45.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:45.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:45.001 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:45.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836581 2024-09-13T16:20:45.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836581 2024-09-13T16:20:45.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581' 2024-09-13T16:20:45.135 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:45.135 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:45.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673057 2024-09-13T16:20:45.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673057 2024-09-13T16:20:45.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673057' 2024-09-13T16:20:45.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:45.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:45.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509534 2024-09-13T16:20:45.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509534 2024-09-13T16:20:45.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836581 1-42949673057 2-64424509534' 2024-09-13T16:20:45.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:45.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836581 2024-09-13T16:20:45.411 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:45.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:45.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836581 2024-09-13T16:20:45.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:45.413 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836581 2024-09-13T16:20:45.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836581 2024-09-13T16:20:45.414 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-09-13T16:20:45.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:45.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836579 -lt 21474836581 2024-09-13T16:20:45.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:20:46.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:20:46.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:47.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836581 -lt 21474836581 2024-09-13T16:20:47.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:47.050 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673057 2024-09-13T16:20:47.051 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:47.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:47.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673057 2024-09-13T16:20:47.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:47.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673057 2024-09-13T16:20:47.055 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673057 2024-09-13T16:20:47.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-09-13T16:20:47.055 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:47.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673058 -lt 42949673057 2024-09-13T16:20:47.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:47.397 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509534 2024-09-13T16:20:47.397 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:47.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:47.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509534 2024-09-13T16:20:47.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:47.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509534 2024-09-13T16:20:47.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509534' 2024-09-13T16:20:47.401 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509534 2024-09-13T16:20:47.401 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:47.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509534 -lt 64424509534 2024-09-13T16:20:47.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:47.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:47.710 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:47.992 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:48.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:48.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:48.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:48.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:48.008 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:48.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:48.315 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:48.315 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:48.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:48.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:48.315 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:48.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836582 2024-09-13T16:20:48.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836582 2024-09-13T16:20:48.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582' 2024-09-13T16:20:48.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:48.452 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:48.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673059 2024-09-13T16:20:48.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673059 2024-09-13T16:20:48.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582 1-42949673059' 2024-09-13T16:20:48.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:48.584 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:48.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509536 2024-09-13T16:20:48.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509536 2024-09-13T16:20:48.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836582 1-42949673059 2-64424509536' 2024-09-13T16:20:48.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:48.716 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:48.716 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836582 2024-09-13T16:20:48.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:48.717 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836582 2024-09-13T16:20:48.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:48.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836582 2024-09-13T16:20:48.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2024-09-13T16:20:48.719 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836582 2024-09-13T16:20:48.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:49.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836582 -lt 21474836582 2024-09-13T16:20:49.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:49.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673059 2024-09-13T16:20:49.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:49.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:49.016 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673059 2024-09-13T16:20:49.016 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:49.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673059 2024-09-13T16:20:49.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-09-13T16:20:49.017 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673059 2024-09-13T16:20:49.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:49.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673059 -lt 42949673059 2024-09-13T16:20:49.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:49.310 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509536 2024-09-13T16:20:49.311 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:49.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:49.313 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509536 2024-09-13T16:20:49.313 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:49.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509536 2024-09-13T16:20:49.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-09-13T16:20:49.315 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509536 2024-09-13T16:20:49.315 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:49.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509535 -lt 64424509536 2024-09-13T16:20:49.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:20:50.625 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:20:50.626 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:50.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509536 -lt 64424509536 2024-09-13T16:20:50.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:50.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:50.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:51.296 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:51.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:51.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:51.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:51.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:51.314 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:51.618 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:51.618 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:51.618 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:51.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:51.619 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:51.619 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:51.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836584 2024-09-13T16:20:51.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836584 2024-09-13T16:20:51.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584' 2024-09-13T16:20:51.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:51.761 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:51.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673061 2024-09-13T16:20:51.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673061 2024-09-13T16:20:51.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673061' 2024-09-13T16:20:51.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:51.887 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:52.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509537 2024-09-13T16:20:52.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509537 2024-09-13T16:20:52.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673061 2-64424509537' 2024-09-13T16:20:52.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:52.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836584 2024-09-13T16:20:52.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:52.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:52.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836584 2024-09-13T16:20:52.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:52.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836584 2024-09-13T16:20:52.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-09-13T16:20:52.016 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836584 2024-09-13T16:20:52.016 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:52.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836583 -lt 21474836584 2024-09-13T16:20:52.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:20:53.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:20:53.319 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:53.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836584 -lt 21474836584 2024-09-13T16:20:53.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:53.631 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:53.631 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673061 2024-09-13T16:20:53.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:53.632 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673061 2024-09-13T16:20:53.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:53.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673061 2024-09-13T16:20:53.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-09-13T16:20:53.634 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673061 2024-09-13T16:20:53.634 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:53.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673061 -lt 42949673061 2024-09-13T16:20:53.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:53.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509537 2024-09-13T16:20:53.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:53.937 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:53.937 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509537 2024-09-13T16:20:53.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:53.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509537 2024-09-13T16:20:53.939 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509537 2024-09-13T16:20:53.939 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-09-13T16:20:53.939 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:54.236 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509537 -lt 64424509537 2024-09-13T16:20:54.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:54.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:54.237 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:54.541 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:54.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:54.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:54.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:54.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:54.560 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:54.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:54.872 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:54.872 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:54.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:54.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:54.872 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:55.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836586 2024-09-13T16:20:55.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836586 2024-09-13T16:20:55.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586' 2024-09-13T16:20:55.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:55.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:55.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673062 2024-09-13T16:20:55.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673062 2024-09-13T16:20:55.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673062' 2024-09-13T16:20:55.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:55.144 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:55.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509539 2024-09-13T16:20:55.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509539 2024-09-13T16:20:55.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836586 1-42949673062 2-64424509539' 2024-09-13T16:20:55.411 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:55.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836586 2024-09-13T16:20:55.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:55.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:55.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836586 2024-09-13T16:20:55.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:55.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836586 2024-09-13T16:20:55.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2024-09-13T16:20:55.413 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836586 2024-09-13T16:20:55.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:55.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836584 -lt 21474836586 2024-09-13T16:20:55.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:20:56.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:20:56.598 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:56.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836586 -lt 21474836586 2024-09-13T16:20:56.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:56.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673062 2024-09-13T16:20:56.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:56.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:56.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673062 2024-09-13T16:20:56.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:56.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673062 2024-09-13T16:20:56.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-09-13T16:20:56.920 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673062 2024-09-13T16:20:56.920 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:57.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673063 -lt 42949673062 2024-09-13T16:20:57.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:57.260 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509539 2024-09-13T16:20:57.260 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:57.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:57.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509539 2024-09-13T16:20:57.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:57.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509539 2024-09-13T16:20:57.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509539' 2024-09-13T16:20:57.264 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509539 2024-09-13T16:20:57.264 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:57.563 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509539 -lt 64424509539 2024-09-13T16:20:57.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:57.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:57.564 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:57.844 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:57.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:57.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:57.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:57.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:57.862 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:20:58.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:20:58.165 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:20:58.165 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:20:58.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:20:58.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:58.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:20:58.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836587 2024-09-13T16:20:58.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836587 2024-09-13T16:20:58.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587' 2024-09-13T16:20:58.334 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:58.334 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:20:58.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673064 2024-09-13T16:20:58.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673064 2024-09-13T16:20:58.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673064' 2024-09-13T16:20:58.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:20:58.470 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:20:58.607 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509541 2024-09-13T16:20:58.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509541 2024-09-13T16:20:58.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673064 2-64424509541' 2024-09-13T16:20:58.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:58.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836587 2024-09-13T16:20:58.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:58.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:20:58.610 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836587 2024-09-13T16:20:58.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:58.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836587 2024-09-13T16:20:58.612 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836587 2024-09-13T16:20:58.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-09-13T16:20:58.613 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:20:58.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836587 -lt 21474836587 2024-09-13T16:20:58.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:58.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673064 2024-09-13T16:20:58.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:58.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:20:58.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673064 2024-09-13T16:20:58.928 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:58.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673064 2024-09-13T16:20:58.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673064' 2024-09-13T16:20:58.931 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673064 2024-09-13T16:20:58.931 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:20:59.228 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673064 -lt 42949673064 2024-09-13T16:20:59.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:20:59.229 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509541 2024-09-13T16:20:59.230 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:20:59.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:20:59.232 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509541 2024-09-13T16:20:59.232 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:20:59.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509541 2024-09-13T16:20:59.233 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509541 2024-09-13T16:20:59.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509541' 2024-09-13T16:20:59.234 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:20:59.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509541 -lt 64424509541 2024-09-13T16:20:59.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:20:59.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:20:59.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:20:59.891 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:20:59.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:20:59.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:20:59.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:20:59.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:20:59.908 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:21:00.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:21:00.230 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:21:00.230 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:21:00.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:21:00.230 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:00.231 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:21:00.417 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836589 2024-09-13T16:21:00.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836589 2024-09-13T16:21:00.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589' 2024-09-13T16:21:00.418 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:00.418 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:21:00.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673065 2024-09-13T16:21:00.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673065 2024-09-13T16:21:00.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673065' 2024-09-13T16:21:00.562 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:00.562 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:21:00.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509542 2024-09-13T16:21:00.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509542 2024-09-13T16:21:00.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836589 1-42949673065 2-64424509542' 2024-09-13T16:21:00.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:00.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836589 2024-09-13T16:21:00.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:00.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:21:00.706 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836589 2024-09-13T16:21:00.706 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:00.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836589 2024-09-13T16:21:00.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-09-13T16:21:00.707 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836589 2024-09-13T16:21:00.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:21:01.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836589 -lt 21474836589 2024-09-13T16:21:01.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:01.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:01.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673065 2024-09-13T16:21:01.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:21:01.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673065 2024-09-13T16:21:01.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:01.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673065 2024-09-13T16:21:01.051 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673065 2024-09-13T16:21:01.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673065' 2024-09-13T16:21:01.051 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:21:01.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673065 -lt 42949673065 2024-09-13T16:21:01.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:01.420 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509542 2024-09-13T16:21:01.421 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:01.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:21:01.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509542 2024-09-13T16:21:01.423 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:01.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509542 2024-09-13T16:21:01.424 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509542 2024-09-13T16:21:01.424 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509542' 2024-09-13T16:21:01.424 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:21:01.730 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509542 -lt 64424509542 2024-09-13T16:21:01.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:21:01.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:21:01.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:21:02.017 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:21:02.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:21:02.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:21:02.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:21:02.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:21:02.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:21:02.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:21:02.351 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:21:02.351 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:21:02.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:21:02.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:02.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:21:02.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836590 2024-09-13T16:21:02.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836590 2024-09-13T16:21:02.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590' 2024-09-13T16:21:02.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:02.488 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:21:02.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673067 2024-09-13T16:21:02.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673067 2024-09-13T16:21:02.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673067' 2024-09-13T16:21:02.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:02.623 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:21:02.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509543 2024-09-13T16:21:02.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509543 2024-09-13T16:21:02.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836590 1-42949673067 2-64424509543' 2024-09-13T16:21:02.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:02.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836590 2024-09-13T16:21:02.763 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:02.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:21:02.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836590 2024-09-13T16:21:02.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:02.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836590 2024-09-13T16:21:02.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836590' 2024-09-13T16:21:02.765 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836590 2024-09-13T16:21:02.766 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:21:03.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836590 -lt 21474836590 2024-09-13T16:21:03.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:03.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:03.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673067 2024-09-13T16:21:03.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:21:03.065 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:03.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673067 2024-09-13T16:21:03.066 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673067 2024-09-13T16:21:03.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673067 2024-09-13T16:21:03.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673067' 2024-09-13T16:21:03.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:21:03.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673067 -lt 42949673067 2024-09-13T16:21:03.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:03.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509543 2024-09-13T16:21:03.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:03.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:21:03.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509543 2024-09-13T16:21:03.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:03.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509543 2024-09-13T16:21:03.404 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509543 2024-09-13T16:21:03.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509543' 2024-09-13T16:21:03.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:21:03.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509542 -lt 64424509543 2024-09-13T16:21:03.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:21:04.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:21:04.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:21:05.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509544 -lt 64424509543 2024-09-13T16:21:05.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:21:05.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:21:05.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:21:05.334 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:21:05.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:21:05.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:21:05.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:21:05.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:21:05.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:21:05.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:21:05.675 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:21:05.675 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:21:05.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:21:05.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:05.676 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:21:05.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836592 2024-09-13T16:21:05.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836592 2024-09-13T16:21:05.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592' 2024-09-13T16:21:05.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:05.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:21:05.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673068 2024-09-13T16:21:05.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673068 2024-09-13T16:21:05.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068' 2024-09-13T16:21:05.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:05.965 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:21:06.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509545 2024-09-13T16:21:06.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509545 2024-09-13T16:21:06.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509545' 2024-09-13T16:21:06.147 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:06.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836592 2024-09-13T16:21:06.148 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:06.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:21:06.149 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836592 2024-09-13T16:21:06.149 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:06.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836592 2024-09-13T16:21:06.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2024-09-13T16:21:06.151 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836592 2024-09-13T16:21:06.151 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:21:06.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836590 -lt 21474836592 2024-09-13T16:21:06.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:21:07.474 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:21:07.474 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:21:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836592 -lt 21474836592 2024-09-13T16:21:07.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:07.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673068 2024-09-13T16:21:07.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:07.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:21:07.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673068 2024-09-13T16:21:07.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:07.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673068 2024-09-13T16:21:07.808 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673068 2024-09-13T16:21:07.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673068' 2024-09-13T16:21:07.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:21:08.174 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673069 -lt 42949673068 2024-09-13T16:21:08.175 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:08.175 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509545 2024-09-13T16:21:08.176 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:08.177 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:21:08.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509545 2024-09-13T16:21:08.177 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:08.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509545 2024-09-13T16:21:08.179 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509545 2024-09-13T16:21:08.179 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509545' 2024-09-13T16:21:08.179 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:21:08.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509545 -lt 64424509545 2024-09-13T16:21:08.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:21:08.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:21:08.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:21:08.805 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:21:08.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:21:08.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:21:08.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:21:08.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:21:08.824 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:21:09.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:21:09.132 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:21:09.132 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:21:09.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:21:09.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:09.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:21:09.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836593 2024-09-13T16:21:09.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836593 2024-09-13T16:21:09.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593' 2024-09-13T16:21:09.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:09.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:21:09.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673070 2024-09-13T16:21:09.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673070 2024-09-13T16:21:09.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673070' 2024-09-13T16:21:09.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:09.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:21:09.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509547 2024-09-13T16:21:09.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509547 2024-09-13T16:21:09.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836593 1-42949673070 2-64424509547' 2024-09-13T16:21:09.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:09.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836593 2024-09-13T16:21:09.513 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:09.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:21:09.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836593 2024-09-13T16:21:09.514 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:09.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836593 2024-09-13T16:21:09.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2024-09-13T16:21:09.515 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836593 2024-09-13T16:21:09.516 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:21:09.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836592 -lt 21474836593 2024-09-13T16:21:09.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:21:10.821 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:21:10.822 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:21:11.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836594 -lt 21474836593 2024-09-13T16:21:11.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:11.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673070 2024-09-13T16:21:11.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:11.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:21:11.164 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673070 2024-09-13T16:21:11.164 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:11.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673070 2024-09-13T16:21:11.165 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673070 2024-09-13T16:21:11.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673070' 2024-09-13T16:21:11.165 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:21:11.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673070 -lt 42949673070 2024-09-13T16:21:11.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:11.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509547 2024-09-13T16:21:11.505 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:11.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:21:11.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509547 2024-09-13T16:21:11.507 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:11.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509547 2024-09-13T16:21:11.509 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509547 2024-09-13T16:21:11.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509547' 2024-09-13T16:21:11.509 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:21:11.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509547 -lt 64424509547 2024-09-13T16:21:11.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:21:11.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:21:11.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:21:12.121 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:21:12.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:21:12.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:21:12.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:21:12.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:21:12.138 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:21:12.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:21:12.463 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:21:12.463 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:21:12.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:21:12.463 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:12.464 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:21:12.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836595 2024-09-13T16:21:12.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836595 2024-09-13T16:21:12.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836595' 2024-09-13T16:21:12.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:12.596 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:21:12.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673072 2024-09-13T16:21:12.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673072 2024-09-13T16:21:12.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836595 1-42949673072' 2024-09-13T16:21:12.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:12.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:21:12.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509548 2024-09-13T16:21:12.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509548 2024-09-13T16:21:12.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836595 1-42949673072 2-64424509548' 2024-09-13T16:21:12.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:12.860 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836595 2024-09-13T16:21:12.860 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:12.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:21:12.862 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836595 2024-09-13T16:21:12.862 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:12.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836595 2024-09-13T16:21:12.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836595' 2024-09-13T16:21:12.864 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836595 2024-09-13T16:21:12.864 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:21:13.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836595 -lt 21474836595 2024-09-13T16:21:13.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:13.219 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673072 2024-09-13T16:21:13.219 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:13.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:21:13.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673072 2024-09-13T16:21:13.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:13.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673072 2024-09-13T16:21:13.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673072' 2024-09-13T16:21:13.223 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673072 2024-09-13T16:21:13.223 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:21:13.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673071 -lt 42949673072 2024-09-13T16:21:13.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:21:14.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:21:14.546 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:21:14.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673072 -lt 42949673072 2024-09-13T16:21:14.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:14.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:14.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509548 2024-09-13T16:21:14.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:21:14.878 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509548 2024-09-13T16:21:14.878 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:14.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509548 2024-09-13T16:21:14.880 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509548 2024-09-13T16:21:14.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509548' 2024-09-13T16:21:14.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:21:15.181 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509549 -lt 64424509548 2024-09-13T16:21:15.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:21:15.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:21:15.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:21:15.485 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:21:15.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: continue 2024-09-13T16:21:15.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: for i in $(seq 0 200) 2024-09-13T16:21:15.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:379: _scrub_abort: flush_pg_stats 2024-09-13T16:21:15.502 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:21:15.502 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:21:15.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:21:15.812 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:21:15.812 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:21:15.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:21:15.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:15.813 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:21:15.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836597 2024-09-13T16:21:15.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836597 2024-09-13T16:21:15.954 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597' 2024-09-13T16:21:15.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:15.955 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:21:16.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673073 2024-09-13T16:21:16.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673073 2024-09-13T16:21:16.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673073' 2024-09-13T16:21:16.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:21:16.134 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:21:16.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509550 2024-09-13T16:21:16.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509550 2024-09-13T16:21:16.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836597 1-42949673073 2-64424509550' 2024-09-13T16:21:16.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:16.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836597 2024-09-13T16:21:16.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:16.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:21:16.264 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836597 2024-09-13T16:21:16.264 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:16.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836597 2024-09-13T16:21:16.265 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836597 2024-09-13T16:21:16.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836597' 2024-09-13T16:21:16.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:21:16.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836595 -lt 21474836597 2024-09-13T16:21:16.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:21:17.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:21:17.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:21:17.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836597 -lt 21474836597 2024-09-13T16:21:17.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:17.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673073 2024-09-13T16:21:17.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:17.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:21:17.900 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673073 2024-09-13T16:21:17.900 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:17.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673073 2024-09-13T16:21:17.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673073' 2024-09-13T16:21:17.901 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949673073 2024-09-13T16:21:17.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:21:18.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673074 -lt 42949673073 2024-09-13T16:21:18.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:21:18.202 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509550 2024-09-13T16:21:18.202 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:21:18.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:21:18.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509550 2024-09-13T16:21:18.204 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:21:18.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509550 2024-09-13T16:21:18.206 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509550 2024-09-13T16:21:18.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509550' 2024-09-13T16:21:18.206 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:21:18.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509550 -lt 64424509550 2024-09-13T16:21:18.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: ceph pg dump pgs 2024-09-13T16:21:18.514 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep '^1.0' 2024-09-13T16:21:18.515 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: grep -q scrubbing 2024-09-13T16:21:18.800 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:21:18.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:385: _scrub_abort: break 2024-09-13T16:21:18.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:387: _scrub_abort: set +o pipefail 2024-09-13T16:21:18.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:389: _scrub_abort: sleep 5 2024-09-13T16:21:23.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:391: _scrub_abort: grep 'noscrub set, aborting' td/osd-scrub-test/osd.1.log 2024-09-13T16:21:23.837 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T16:19:20.877+0000 7f1bd41a1640 10 osd.1 pg_epoch: 20 pg[1.0( v 19'1000 (0'0,19'1000] local-lis/les=17/18 n=1000 ec=17/17 lis/c=17/17 les/c/f=18/18/0 sis=17) [1,0,2] r=0 lpr=17 crt=19'1000 lcod 19'999 mlcod 19'999 active+clean+scrubbing [ 1.0: ] ] scrubber: noscrub set, aborting 2024-09-13T16:21:23.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:397: _scrub_abort: get_last_scrub_stamp 1.0 2024-09-13T16:21:23.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:23.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:23.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:23.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:24.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:397: _scrub_abort: local last_scrub=2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:24.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:398: _scrub_abort: ceph config set osd osd_scrub_sleep 0.1 2024-09-13T16:21:24.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:400: _scrub_abort: ceph osd unset noscrub 2024-09-13T16:21:25.242 INFO:tasks.workunit.client.0.smithi112.stderr:noscrub is unset 2024-09-13T16:21:25.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:401: _scrub_abort: '[' scrub = deep-scrub ']' 2024-09-13T16:21:25.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:405: _scrub_abort: TIMEOUT=500 2024-09-13T16:21:25.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:406: _scrub_abort: wait_for_scrub 1.0 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:25.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T16:21:25.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:25.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T16:21:25.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T16:21:25.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:25.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:25.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:25.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:25.266 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:25.267 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:25.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:25.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:26.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:26.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:26.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:26.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:26.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:26.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:26.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:26.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:26.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:27.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:27.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:27.922 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:27.922 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:27.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:27.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:27.923 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:28.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:28.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:29.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:29.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:29.252 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:29.252 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:29.252 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:29.252 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:29.253 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:29.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:29.565 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:30.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:30.566 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:30.567 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:30.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:30.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:30.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:30.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:30.879 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:30.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:31.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:31.881 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:31.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:31.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:31.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:31.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:31.882 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:32.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:32.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:33.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:33.204 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:33.205 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:33.205 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:33.205 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:33.205 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:33.206 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:33.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:33.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:34.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:34.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:34.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:34.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:34.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:34.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:34.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:34.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:34.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:35.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:35.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:35.836 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:35.836 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:35.836 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:35.837 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:35.837 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:36.148 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:36.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:37.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:37.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:37.150 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:37.150 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:37.150 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:37.151 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:37.151 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:37.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:37.460 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:38.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:38.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:38.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:38.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:38.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:38.463 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:38.464 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:38.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:38.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:39.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:39.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:39.789 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:39.789 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:39.789 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:39.789 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:39.789 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:40.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:40.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:41.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:41.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:41.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:41.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:41.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:41.119 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:41.119 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:41.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:41.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:42.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:42.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:42.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:42.456 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:42.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:42.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:42.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:42.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:42.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:43.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:43.783 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:43.783 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:43.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:43.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:43.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:43.784 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:44.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:44.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:45.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:45.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:45.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:45.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:45.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:45.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:45.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:45.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:45.412 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:46.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:46.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:46.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:46.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:46.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:46.415 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:46.415 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:46.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:46.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:47.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:47.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:47.729 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:47.729 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:47.729 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:47.729 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:47.729 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:48.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:48.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:49.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:49.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:49.059 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:49.059 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:49.059 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:49.060 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:49.060 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:49.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:49.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:50.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:50.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:50.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:50.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:50.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:50.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:50.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:50.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:50.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:51.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:51.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:51.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:51.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:51.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:51.711 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:51.711 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:52.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:52.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:53.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:53.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:53.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:53.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:53.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:53.042 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:53.042 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:53.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:53.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:54.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:54.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:54.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:54.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:54.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:54.409 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:54.410 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:54.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:54.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:55.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:55.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:55.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:55.721 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:55.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:55.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:55.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:56.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:56.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:57.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:57.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:57.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:57.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:57.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:57.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:57.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:57.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:57.381 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:58.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:58.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:58.383 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:58.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:58.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:58.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:58.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:21:58.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:21:58.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:21:59.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:21:59.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:21:59.717 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:21:59.717 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:21:59.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:21:59.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:21:59.718 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:00.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:00.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:01.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:01.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:01.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:01.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:01.041 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:01.042 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:01.042 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:01.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:01.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:02.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:02.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:02.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:02.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:02.357 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:02.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:02.358 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:02.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:02.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:03.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:03.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:03.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:03.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:03.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:03.684 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:03.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:03.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:03.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:05.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:05.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:05.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:05.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:05.002 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:05.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:05.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:05.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:05.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:06.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:06.352 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:06.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:06.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:06.353 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:06.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:06.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:06.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:06.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:07.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:07.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:07.677 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:07.677 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:07.677 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:07.678 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:07.678 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:07.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:07.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:08.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:08.993 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:08.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:08.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:08.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:08.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:08.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:09.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:09.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:10.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:10.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:10.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:10.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:10.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:10.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:10.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:10.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:10.634 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:11.635 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:11.636 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:11.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:11.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:11.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:11.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:11.637 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:11.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:11.963 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:12.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:12.964 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:12.965 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:12.965 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:12.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:12.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:12.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:13.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:13.321 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:14.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:14.322 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:14.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:14.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:14.323 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:14.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:14.324 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:14.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:14.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:15.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:15.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:15.642 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:15.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:15.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:15.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:15.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:15.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:15.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:16.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:16.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:16.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:16.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:16.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:16.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:16.950 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:17.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:17.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:18.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:18.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:18.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:18.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:18.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:18.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:18.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:18.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:18.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:19.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:19.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:19.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:19.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:19.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:19.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:19.595 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:19.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:19.922 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:20.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:20.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:20.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:20.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:20.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:20.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:20.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:21.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:21.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:22.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:22.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:22.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:22.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:22.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:22.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:22.246 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:22.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:22.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:23.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:23.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:23.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:23.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:23.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:23.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:23.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:23.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:23.882 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:24.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:24.884 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:24.884 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:24.884 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:24.884 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:24.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:24.885 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:25.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:25.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:26.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:26.208 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:26.208 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:26.208 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:26.208 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:26.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:26.209 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:26.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-06T16:17:34.904618+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:26.542 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:22:27.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:22:27.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:22:27.544 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:22:27.544 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:22:27.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:22:27.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:22:27.545 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:22:27.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:22:22.668099+0000 '>' 2024-09-06T16:17:34.904618+0000 2024-09-13T16:22:27.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T16:22:27.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:407: _scrub_abort: perf_counters td/osd-scrub-test 3 2024-09-13T16:22:27.860 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-09-13T16:22:27.861 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2024-09-13T16:22:27.861 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2024-09-13T16:22:27.862 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2024-09-13T16:22:27.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:22:27.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-09-13T16:22:27.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:22:27.987 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:22:27.987 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:22:27.987 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:22:27.987 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:22:27.987 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:22:27.988 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:22:27.988 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.988 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:22:27.988 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:22:27.988 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:22:27.988 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:22:27.988 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:22:27.988 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:27.988 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:27.988 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:27.988 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.988 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:22:27.989 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:22:27.989 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:27.989 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:27.989 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:27.989 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.989 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:22:27.989 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:22:27.989 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:22:27.989 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:22:27.989 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:22:27.989 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:22:27.989 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:22:27.990 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:27.990 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:27.990 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:27.990 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.990 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:22:27.990 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:22:27.990 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:22:27.990 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:22:27.990 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:27.990 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:27.990 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:27.990 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.990 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:22:27.990 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:27.991 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:27.991 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:22:27.991 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:22:27.991 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:22:27.991 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:22:27.991 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:22:27.991 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:22:27.991 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.991 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:22:27.991 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:22:27.991 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:22:27.991 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:22:27.991 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:22:27.992 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:27.992 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:27.992 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:27.992 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.992 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:22:27.992 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:22:27.992 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:27.992 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:27.992 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:27.992 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.992 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:22:27.992 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:22:27.992 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:22:27.993 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:22:27.993 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:22:27.993 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:22:27.993 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:22:27.993 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:27.993 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:27.993 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:27.993 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.993 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:22:27.993 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:22:27.993 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:22:27.993 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:22:27.993 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:27.994 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:27.994 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:27.994 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.994 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:22:27.994 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:27.994 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:27.994 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:22:27.994 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:22:27.994 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:22:27.994 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:22:27.994 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:22:27.994 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:22:27.994 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.994 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:22:27.995 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:22:27.995 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:22:27.995 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:22:27.995 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:22:27.995 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:27.995 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:27.995 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:27.995 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.995 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:22:27.995 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:22:27.995 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:27.995 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:27.995 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:27.995 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.996 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:22:27.996 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:22:27.996 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:22:27.996 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:22:27.996 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:22:27.996 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:22:27.996 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:22:27.996 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:27.996 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:27.996 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:27.996 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.996 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:22:27.996 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:22:27.997 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:22:27.997 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:22:27.997 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:27.997 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:27.997 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:27.997 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.997 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:22:27.997 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:27.997 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:27.997 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:22:27.997 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:22:27.997 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:22:27.997 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:22:27.998 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:22:27.998 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:22:27.998 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.998 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:22:27.998 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:22:27.998 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:22:27.998 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:22:27.998 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:22:27.998 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:27.998 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:27.998 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:27.998 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.998 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:22:27.999 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:22:27.999 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:27.999 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:27.999 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:27.999 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:27.999 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:22:27.999 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:22:27.999 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:22:27.999 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:22:27.999 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:22:27.999 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:22:27.999 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:22:28.000 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.000 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.000 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.000 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.000 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:22:28.000 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:22:28.000 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:22:28.000 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:22:28.000 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.000 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.000 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.001 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.001 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:22:28.001 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.001 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.001 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:22:28.001 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:22:28.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:22:28.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-09-13T16:22:28.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:22:28.155 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:22:28.155 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:22:28.155 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:22:28.155 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:22:28.155 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:22:28.155 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:22:28.156 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.156 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:22:28.156 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:22:28.156 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:22:28.156 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:22:28.156 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:22:28.156 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.156 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.156 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.156 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.156 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:22:28.156 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:22:28.156 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.157 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.157 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.157 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.157 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:22:28.157 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:22:28.157 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:22:28.157 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:22:28.157 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:22:28.157 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:22:28.157 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:22:28.157 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.157 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.157 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.158 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.158 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:22:28.158 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:22:28.158 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:22:28.158 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:22:28.158 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.158 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.158 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.158 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.158 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:22:28.158 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.158 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.158 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:22:28.158 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:22:28.159 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:22:28.159 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:22:28.159 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:22:28.159 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:22:28.159 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.159 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:22:28.159 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:22:28.159 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:22:28.159 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:22:28.159 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:22:28.159 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.159 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.159 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.160 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.160 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:22:28.160 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:22:28.160 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.160 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.160 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.160 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.160 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:22:28.160 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:22:28.160 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:22:28.160 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:22:28.160 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:22:28.160 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:22:28.161 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:22:28.161 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.161 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.161 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.161 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.161 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:22:28.161 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:22:28.161 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:22:28.161 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:22:28.161 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.161 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.161 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.161 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.161 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:22:28.162 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.162 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.162 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:22:28.162 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:22:28.162 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:22:28.162 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:22:28.162 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:22:28.162 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:22:28.162 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.162 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:22:28.162 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:22:28.162 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:22:28.162 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:22:28.162 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:22:28.163 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.163 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.163 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.163 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.163 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:22:28.163 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:22:28.163 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.163 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.163 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.163 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.163 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:22:28.163 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:22:28.163 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:22:28.163 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:22:28.164 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:22:28.164 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:22:28.164 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:22:28.164 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.164 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.164 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.164 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.164 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:22:28.164 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:22:28.164 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:22:28.164 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:22:28.164 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.164 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.165 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.165 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.165 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:22:28.165 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.165 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.165 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:22:28.165 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:22:28.165 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:22:28.165 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:22:28.165 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:22:28.165 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:22:28.165 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.165 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:22:28.165 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 2, 2024-09-13T16:22:28.166 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 2, 2024-09-13T16:22:28.166 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 1, 2024-09-13T16:22:28.166 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:22:28.166 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 1, 2024-09-13T16:22:28.166 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 55.230593927, 2024-09-13T16:22:28.166 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 55.230593927 2024-09-13T16:22:28.166 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.166 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 1, 2024-09-13T16:22:28.166 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:22:28.166 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 1, 2024-09-13T16:22:28.166 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 5.003872302, 2024-09-13T16:22:28.166 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 5.003872302 2024-09-13T16:22:28.166 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.166 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:22:28.167 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 11, 2024-09-13T16:22:28.167 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:22:28.167 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:22:28.167 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:22:28.167 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 2, 2024-09-13T16:22:28.168 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:22:28.168 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 2, 2024-09-13T16:22:28.168 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0.003999899, 2024-09-13T16:22:28.168 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0.001999949 2024-09-13T16:22:28.168 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.168 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:22:28.168 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:22:28.168 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:22:28.168 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:22:28.169 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.169 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.169 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.169 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.169 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 2 2024-09-13T16:22:28.169 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.169 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.169 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:22:28.169 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:22:28.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:22:28.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2024-09-13T16:22:28.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:22:28.283 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:22:28.283 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:22:28.284 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:22:28.284 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:22:28.284 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:22:28.284 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:22:28.284 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.284 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:22:28.284 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:22:28.284 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:22:28.284 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:22:28.284 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:22:28.284 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.284 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.285 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.285 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.285 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:22:28.285 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:22:28.285 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.285 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.285 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.285 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.285 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:22:28.285 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:22:28.286 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:22:28.286 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:22:28.286 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:22:28.286 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:22:28.286 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:22:28.286 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.286 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.286 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.286 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.286 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:22:28.286 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:22:28.286 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:22:28.286 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:22:28.287 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.287 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.287 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.287 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.287 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:22:28.287 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.287 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.287 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:22:28.287 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:22:28.287 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:22:28.288 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:22:28.288 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:22:28.288 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:22:28.288 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.288 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:22:28.288 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:22:28.288 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:22:28.288 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:22:28.288 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:22:28.288 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.288 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.288 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.289 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.289 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:22:28.289 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:22:28.289 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.289 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.289 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.289 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.289 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:22:28.289 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:22:28.289 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:22:28.289 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:22:28.290 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:22:28.290 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:22:28.290 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:22:28.290 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.290 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.290 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.290 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.290 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:22:28.290 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:22:28.290 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:22:28.290 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:22:28.290 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.290 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.291 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.291 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.291 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:22:28.291 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.292 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.292 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:22:28.292 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:22:28.292 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:22:28.292 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:22:28.292 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:22:28.292 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:22:28.292 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.292 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:22:28.293 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:22:28.293 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:22:28.293 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:22:28.293 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:22:28.293 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.293 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.293 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.293 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.293 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:22:28.293 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:22:28.293 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.294 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.294 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.294 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.294 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:22:28.294 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:22:28.294 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:22:28.294 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:22:28.294 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:22:28.294 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:22:28.294 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:22:28.295 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.295 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.295 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.295 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.295 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:22:28.295 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:22:28.295 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:22:28.295 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:22:28.295 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.295 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.295 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.295 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.296 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:22:28.296 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.296 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.296 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:22:28.296 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:22:28.296 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:22:28.296 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:22:28.296 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:22:28.297 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:22:28.297 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.297 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:22:28.297 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:22:28.297 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:22:28.297 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:22:28.298 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:22:28.298 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.298 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.298 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.299 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.299 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:22:28.299 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:22:28.299 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.299 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.299 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.300 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.300 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:22:28.300 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:22:28.300 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:22:28.300 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:22:28.300 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:22:28.300 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:22:28.300 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:22:28.301 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.301 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.301 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.301 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.301 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:22:28.301 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:22:28.301 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:22:28.301 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:22:28.302 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:22:28.302 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:22:28.302 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:22:28.302 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:22:28.302 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:22:28.302 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.302 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:22:28.302 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:22:28.302 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:22:28.303 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_scrub_abort ------------------' 2024-09-13T16:22:28.303 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Teardown Test TEST_scrub_abort ------------------ 2024-09-13T16:22:28.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-09-13T16:22:28.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:22:28.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:22:28.304 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:22:28.305 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:22:28.305 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:22:28.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:22:28.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:22:28.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:22:28.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:22:28.605 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:22:28.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:22:28.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:22:28.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:22:28.608 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:22:28.609 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:22:28.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:22:28.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:22:28.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:22:28.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:22:28.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:22:28.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:22:28.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:22:28.675 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:22:28.675 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:22:28.675 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:22:28.675 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:22:28.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:22:28.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:22:28.677 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Complete Test TEST_scrub_abort ------------------ 2024-09-13T16:22:28.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_scrub_abort ------------------' 2024-09-13T16:22:28.678 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-13T16:22:28.678 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Prepare Test TEST_scrub_permit_time ------------------- 2024-09-13T16:22:28.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_scrub_permit_time -------------------' 2024-09-13T16:22:28.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-09-13T16:22:28.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2024-09-13T16:22:28.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2024-09-13T16:22:28.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:22:28.680 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:22:28.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:22:28.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:22:28.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:22:28.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:22:28.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:22:28.681 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:22:28.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:22:28.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:22:28.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:22:28.683 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:22:28.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:22:28.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:22:28.685 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:22:28.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:22:28.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:22:28.686 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:22:28.687 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:22:28.687 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:22:28.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:22:28.689 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:22:28.690 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:22:28.690 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:22:28.690 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:22:28.690 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:22:28.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:22:28.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:22:28.691 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2024-09-13T16:22:28.693 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T16:22:28.693 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:22:28.693 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:22:28.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.528094 2024-09-13T16:22:28.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T16:22:28.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T16:22:28.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T16:22:28.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-13T16:22:28.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_scrub_permit_time -----------------------' 2024-09-13T16:22:28.695 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Run Test TEST_scrub_permit_time ----------------------- 2024-09-13T16:22:28.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_scrub_permit_time td/osd-scrub-test 2024-09-13T16:22:28.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:421: TEST_scrub_permit_time: local dir=td/osd-scrub-test 2024-09-13T16:22:28.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:422: TEST_scrub_permit_time: local poolname=test 2024-09-13T16:22:28.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:423: TEST_scrub_permit_time: local OSDS=3 2024-09-13T16:22:28.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:424: TEST_scrub_permit_time: local objects=15 2024-09-13T16:22:28.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:426: TEST_scrub_permit_time: TESTDATA=testdata.528094 2024-09-13T16:22:28.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:428: TEST_scrub_permit_time: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-13T16:22:28.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2024-09-13T16:22:28.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T16:22:28.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T16:22:28.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T16:22:28.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2024-09-13T16:22:28.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-13T16:22:29.220 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T16:22:29.220 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:22:29.220 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:22:29.221 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:22:29.221 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:22:29.221 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:22:29.221 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:22:29.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-13T16:22:29.454 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T16:22:29.455 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T16:22:29.455 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T16:22:29.455 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T16:22:29.455 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T16:22:29.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T16:22:29.457 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T16:22:29.457 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T16:22:29.457 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T16:22:29.457 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:22:29.457 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:22:29.457 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:22:29.457 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-mon.a.asok 2024-09-13T16:22:29.457 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T16:22:29.458 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.528094/ceph-mon.a.asok config get fsid 2024-09-13T16:22:29.567 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T16:22:29.567 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T16:22:29.567 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T16:22:29.567 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T16:22:29.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T16:22:29.568 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T16:22:29.568 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T16:22:29.568 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T16:22:29.569 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:22:29.569 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:22:29.569 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:22:29.569 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-mon.a.asok 2024-09-13T16:22:29.569 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T16:22:29.569 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.528094/ceph-mon.a.asok config get mon_host 2024-09-13T16:22:29.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:429: TEST_scrub_permit_time: run_mgr td/osd-scrub-test x 2024-09-13T16:22:29.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2024-09-13T16:22:29.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T16:22:29.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T16:22:29.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T16:22:29.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2024-09-13T16:22:29.670 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T16:22:30.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T16:22:30.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:22:30.607 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:22:30.607 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:22:30.607 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:22:30.607 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:22:30.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:22:30.608 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T16:22:30.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T16:22:30.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:430: TEST_scrub_permit_time: date -d '2 hour ago' +%H 2024-09-13T16:22:30.643 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:430: TEST_scrub_permit_time: sed 's/^0//' 2024-09-13T16:22:30.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:430: TEST_scrub_permit_time: local scrub_begin_hour=14 2024-09-13T16:22:30.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:431: TEST_scrub_permit_time: date -d '1 hour ago' +%H 2024-09-13T16:22:30.645 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:431: TEST_scrub_permit_time: sed 's/^0//' 2024-09-13T16:22:30.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:431: TEST_scrub_permit_time: local scrub_end_hour=15 2024-09-13T16:22:30.647 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:432: TEST_scrub_permit_time: expr 3 - 1 2024-09-13T16:22:30.648 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:432: TEST_scrub_permit_time: seq 0 2 2024-09-13T16:22:30.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:432: TEST_scrub_permit_time: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:22:30.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:434: TEST_scrub_permit_time: run_osd td/osd-scrub-test 0 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 2024-09-13T16:22:30.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:22:30.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:22:30.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T16:22:30.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:22:30.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-13T16:22:30.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:22:30.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:22:30.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:22:30.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:22:30.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-13T16:22:30.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-13T16:22:30.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:22:30.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:22:30.653 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:22:30.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:22:30.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:22:30.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:22:30.654 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:22:30.654 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:22:30.654 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:22:30.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:22:30.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:22:30.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:22:30.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:22:30.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:22:30.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:22:30.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:22:30.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:22:30.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:22:30.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:22:30.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:22:30.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:22:30.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15' 2024-09-13T16:22:30.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-13T16:22:30.655 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:22:30.655 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0e39e9df-4e11-4a3e-96ab-abf38bd11dc0 2024-09-13T16:22:30.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0e39e9df-4e11-4a3e-96ab-abf38bd11dc0' 2024-09-13T16:22:30.656 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 0e39e9df-4e11-4a3e-96ab-abf38bd11dc0 2024-09-13T16:22:30.656 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:22:30.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDGZuRmQPDtJxAAQSSUYsj92qroj/D1zGbqZA== 2024-09-13T16:22:30.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDGZuRmQPDtJxAAQSSUYsj92qroj/D1zGbqZA=="}' 2024-09-13T16:22:30.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0e39e9df-4e11-4a3e-96ab-abf38bd11dc0 -i td/osd-scrub-test/0/new.json 2024-09-13T16:22:30.959 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:22:30.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-13T16:22:30.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 --mkfs --key AQDGZuRmQPDtJxAAQSSUYsj92qroj/D1zGbqZA== --osd-uuid 0e39e9df-4e11-4a3e-96ab-abf38bd11dc0 2024-09-13T16:22:31.009 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:22:31.004+0000 7f06fff447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:22:31.044 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:22:31.040+0000 7f06fff447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:22:31.045 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:22:31.040+0000 7f06fff447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:22:31.045 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:22:31.040+0000 7f06fff447c0 -1 bdev(0x559854c45c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2024-09-13T16:22:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:22:31.040+0000 7f06fff447c0 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-13T16:22:39.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-13T16:22:39.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:22:39.127 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T16:22:39.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T16:22:39.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:22:39.520 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T16:22:39.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T16:22:39.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:22:39.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:22:39.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:22:39.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 2024-09-13T16:22:39.554 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:22:39.548+0000 7f3b0db8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:22:39.591 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:22:39.588+0000 7f3b0db8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:22:39.593 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:22:39.588+0000 7f3b0db8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:22:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T16:22:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:22:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T16:22:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:22:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:22:39.825 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:22:39.826 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:22:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:22:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:22:39.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:22:40.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:22:40.657 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:22:40.652+0000 7f3b0db8e7c0 -1 Falling back to public interface 2024-09-13T16:22:41.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:22:41.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:22:41.139 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:22:41.139 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:22:41.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:22:41.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:22:41.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:22:42.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:22:42.469 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:22:42.470 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:22:42.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:22:42.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:22:42.470 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:22:42.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:22:43.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:22:43.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:22:43.791 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:22:43.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:22:43.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:22:43.791 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:22:43.933 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:22:43.932+0000 7f3b0db8e7c0 -1 osd.0 0 log_to_monitors true 2024-09-13T16:22:44.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:22:45.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:22:45.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:22:45.125 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:22:45.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:22:45.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:22:45.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:22:45.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:22:46.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:22:46.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:22:46.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:22:46.577 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:22:46.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:22:46.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:22:47.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:22:48.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:22:48.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:22:48.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T16:22:48.102 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:22:48.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:22:48.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:22:48.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:22:49.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:22:49.426 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:22:49.426 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:22:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T16:22:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:22:49.427 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:22:49.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:22:50.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:22:50.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:22:50.804 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:22:50.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T16:22:50.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:22:50.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:22:51.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:22:52.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:22:52.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:22:52.270 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:22:52.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T16:22:52.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:22:52.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:22:52.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:22:53.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:22:53.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:22:53.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T16:22:53.586 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:22:53.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:22:53.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:22:54.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:22:55.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:22:55.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:22:55.090 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:22:55.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T16:22:55.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:22:55.090 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:22:55.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:22:56.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:22:56.464 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:22:56.464 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:22:56.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T16:22:56.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:22:56.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:22:56.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:22:57.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:22:57.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:22:57.799 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:22:57.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T16:22:57.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:22:57.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:22:58.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:22:59.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:22:59.284 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:22:59.285 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:22:59.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T16:22:59.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:22:59.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:22:59.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:00.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:00.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:00.839 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:23:00.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T16:23:00.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:00.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:23:01.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:02.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:02.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:02.261 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:23:02.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T16:23:02.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:02.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:23:02.596 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:03.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:03.598 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:03.599 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:23:03.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T16:23:03.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:03.600 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:23:04.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:05.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:05.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:05.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T16:23:05.375 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:23:05.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:05.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:23:06.161 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:07.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:07.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:07.163 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:23:07.163 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T16:23:07.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:07.164 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:23:08.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:09.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:09.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:09.308 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:23:09.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T16:23:09.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:09.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:23:10.323 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:11.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:11.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:11.325 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:23:11.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T16:23:11.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:11.326 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:23:12.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:13.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:13.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:13.047 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:23:13.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T16:23:13.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:13.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:23:13.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:14.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:14.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:14.764 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:23:14.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T16:23:14.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:14.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:23:15.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:16.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:16.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:16.307 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:23:16.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T16:23:16.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:16.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:23:16.719 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1263568628,v1:127.0.0.1:6803/1263568628] [v2:127.0.0.1:6804/1263568628,v1:127.0.0.1:6805/1263568628] exists,up 0e39e9df-4e11-4a3e-96ab-abf38bd11dc0 2024-09-13T16:23:16.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:23:16.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:23:16.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:23:16.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:432: TEST_scrub_permit_time: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:23:16.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:434: TEST_scrub_permit_time: run_osd td/osd-scrub-test 1 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 2024-09-13T16:23:16.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:23:16.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:23:16.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T16:23:16.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:23:16.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-13T16:23:16.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:23:16.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:23:16.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:23:16.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:23:16.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-13T16:23:16.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-13T16:23:16.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:23:16.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:23:16.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:23:16.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:23:16.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:23:16.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:23:16.723 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:23:16.723 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:23:16.723 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:23:16.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:23:16.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:23:16.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:23:16.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:23:16.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:23:16.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:23:16.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:23:16.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:23:16.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:23:16.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:23:16.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:23:16.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:23:16.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15' 2024-09-13T16:23:16.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-13T16:23:16.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:23:16.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f466ed9d-d91f-4d3c-aafa-65d3fd7f0301 2024-09-13T16:23:16.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f466ed9d-d91f-4d3c-aafa-65d3fd7f0301' 2024-09-13T16:23:16.726 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 f466ed9d-d91f-4d3c-aafa-65d3fd7f0301 2024-09-13T16:23:16.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:23:16.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD0ZuRmRo5ZLBAAG0Z5pULS9WRM6lr/YOO3zw== 2024-09-13T16:23:16.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD0ZuRmRo5ZLBAAG0Z5pULS9WRM6lr/YOO3zw=="}' 2024-09-13T16:23:16.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f466ed9d-d91f-4d3c-aafa-65d3fd7f0301 -i td/osd-scrub-test/1/new.json 2024-09-13T16:23:17.380 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:23:17.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-13T16:23:17.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 --mkfs --key AQD0ZuRmRo5ZLBAAG0Z5pULS9WRM6lr/YOO3zw== --osd-uuid f466ed9d-d91f-4d3c-aafa-65d3fd7f0301 2024-09-13T16:23:17.440 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:23:17.439+0000 7fba9abae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:23:17.516 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:23:17.515+0000 7fba9abae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:23:17.518 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:23:17.515+0000 7fba9abae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:23:17.518 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:23:17.519+0000 7fba9abae7c0 -1 bdev(0x558813f87c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2024-09-13T16:23:17.518 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:23:17.519+0000 7fba9abae7c0 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-13T16:23:26.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-13T16:23:26.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:23:26.400 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T16:23:26.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T16:23:26.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:23:26.799 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T16:23:26.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T16:23:26.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:23:26.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:23:26.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:23:26.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 2024-09-13T16:23:26.833 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:23:26.831+0000 7f62032727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:23:26.871 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:23:26.871+0000 7f62032727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:23:26.873 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:23:26.871+0000 7f62032727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:23:27.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T16:23:27.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:23:27.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T16:23:27.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:23:27.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:23:27.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:27.126 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:23:27.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:23:27.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:27.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:27.481 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:27.929 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:23:27.927+0000 7f62032727c0 -1 Falling back to public interface 2024-09-13T16:23:28.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:28.483 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:28.483 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:23:28.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:23:28.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:28.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:28.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:29.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:29.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:29.796 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:23:29.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:23:29.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:29.796 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:30.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:31.041 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:23:31.039+0000 7f62032727c0 -1 osd.1 0 log_to_monitors true 2024-09-13T16:23:31.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:31.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:31.131 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:23:31.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:23:31.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:31.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:31.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:32.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:32.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:32.594 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:23:32.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:23:32.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:32.594 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:32.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:33.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:33.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:33.977 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:23:33.977 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:23:33.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:33.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:34.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:35.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:35.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:35.662 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:23:35.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T16:23:35.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:35.663 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:35.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:36.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:36.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:36.988 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:23:36.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T16:23:36.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:36.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:37.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:38.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:38.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:38.506 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T16:23:38.506 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:23:38.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:38.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:38.822 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:39.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:39.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:39.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T16:23:39.824 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:23:39.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:39.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:40.293 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:41.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:41.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:41.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T16:23:41.296 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:23:41.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:41.296 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:41.738 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:42.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:42.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:42.740 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:23:42.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T16:23:42.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:42.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:43.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:44.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:44.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:44.074 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:23:44.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T16:23:44.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:44.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:44.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:45.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:45.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:45.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T16:23:45.522 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:23:45.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:45.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:45.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:46.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:46.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:46.875 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:23:46.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T16:23:46.875 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:46.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:47.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:48.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:48.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:48.307 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:23:48.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T16:23:48.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:48.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:48.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:49.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:49.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:49.631 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:23:49.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T16:23:49.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:49.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:49.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:50.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:50.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:50.946 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:23:50.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T16:23:50.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:50.947 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:51.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:52.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:52.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:52.626 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:23:52.626 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T16:23:52.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:52.627 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:53.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:54.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:54.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:54.219 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:23:54.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T16:23:54.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:54.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:54.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:55.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:55.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:55.974 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:23:55.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T16:23:55.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:55.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:56.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:57.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:57.759 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:57.760 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:23:57.760 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T16:23:57.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:57.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:23:58.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:23:59.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:23:59.551 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:23:59.552 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:23:59.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T16:23:59.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:23:59.552 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:24:00.325 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:01.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:01.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:01.327 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:24:01.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T16:24:01.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:01.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:24:02.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:03.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:03.038 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:03.038 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:24:03.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T16:24:03.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:03.039 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:24:03.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:04.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:04.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:04.918 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T16:24:04.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T16:24:04.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:04.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:24:05.310 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/138677360,v1:127.0.0.1:6811/138677360] [v2:127.0.0.1:6812/138677360,v1:127.0.0.1:6813/138677360] exists,up f466ed9d-d91f-4d3c-aafa-65d3fd7f0301 2024-09-13T16:24:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:24:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:24:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:24:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:432: TEST_scrub_permit_time: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:24:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:434: TEST_scrub_permit_time: run_osd td/osd-scrub-test 2 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 2024-09-13T16:24:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:24:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:24:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T16:24:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:24:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-13T16:24:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:24:05.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:24:05.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:24:05.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:24:05.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-13T16:24:05.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-13T16:24:05.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:24:05.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:24:05.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:24:05.313 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:24:05.313 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:24:05.313 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:24:05.314 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:24:05.314 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:24:05.315 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:24:05.315 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:24:05.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:24:05.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:24:05.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:24:05.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:24:05.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:24:05.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:24:05.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:24:05.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:24:05.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:24:05.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:24:05.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:24:05.315 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15' 2024-09-13T16:24:05.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-13T16:24:05.316 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:24:05.316 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9ef8cbfa-82b4-4132-af1e-40a5fb7744c2 2024-09-13T16:24:05.316 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 9ef8cbfa-82b4-4132-af1e-40a5fb7744c2 2024-09-13T16:24:05.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 9ef8cbfa-82b4-4132-af1e-40a5fb7744c2' 2024-09-13T16:24:05.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:24:05.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAlZ+RmdNogFBAAMBOL47sHLw2C5hy7tqaBnw== 2024-09-13T16:24:05.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAlZ+RmdNogFBAAMBOL47sHLw2C5hy7tqaBnw=="}' 2024-09-13T16:24:05.336 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9ef8cbfa-82b4-4132-af1e-40a5fb7744c2 -i td/osd-scrub-test/2/new.json 2024-09-13T16:24:06.023 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:24:06.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-13T16:24:06.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 --mkfs --key AQAlZ+RmdNogFBAAMBOL47sHLw2C5hy7tqaBnw== --osd-uuid 9ef8cbfa-82b4-4132-af1e-40a5fb7744c2 2024-09-13T16:24:06.089 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:24:06.086+0000 7f25518da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:24:06.188 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:24:06.186+0000 7f25518da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:24:06.190 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:24:06.186+0000 7f25518da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:24:06.190 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:24:06.190+0000 7f25518da7c0 -1 bdev(0x5574c5837c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2024-09-13T16:24:06.190 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:24:06.190+0000 7f25518da7c0 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-13T16:24:14.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-13T16:24:14.868 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:24:14.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T16:24:14.869 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T16:24:14.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:24:15.267 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T16:24:15.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T16:24:15.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=14 --osd_scrub_end_hour=15 2024-09-13T16:24:15.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:24:15.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:24:15.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:24:15.300 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:24:15.298+0000 7f71564447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:24:15.339 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:24:15.338+0000 7f71564447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:24:15.340 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:24:15.338+0000 7f71564447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:24:15.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T16:24:15.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:24:15.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T16:24:15.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:24:15.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:24:15.588 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:24:15.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:15.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:24:15.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:15.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:15.908 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:16.413 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:24:16.410+0000 7f71564447c0 -1 Falling back to public interface 2024-09-13T16:24:16.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:16.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:16.910 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:24:16.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:24:16.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:16.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:17.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:18.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:18.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:18.225 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:24:18.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:24:18.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:18.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:18.543 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:19.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:19.544 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:19.545 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:24:19.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:24:19.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:19.545 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:19.578 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:24:19.578+0000 7f71564447c0 -1 osd.2 0 log_to_monitors true 2024-09-13T16:24:20.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:21.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:21.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:21.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:24:21.033 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:24:21.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:21.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:21.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:22.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:22.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:22.388 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:24:22.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:24:22.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:22.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:22.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:23.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:23.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:23.916 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:24:23.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T16:24:23.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:23.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:24.401 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:25.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:25.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:25.403 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:24:25.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T16:24:25.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:25.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:25.739 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:26.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:26.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:26.741 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:24:26.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T16:24:26.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:26.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:27.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:28.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:28.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:28.108 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:24:28.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T16:24:28.108 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:28.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:28.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:29.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:29.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:29.448 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:24:29.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T16:24:29.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:29.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:29.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:30.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:30.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:30.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T16:24:30.771 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:24:30.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:30.771 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:31.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:32.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:32.142 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:32.143 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:24:32.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T16:24:32.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:32.143 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:32.656 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:33.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:33.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:33.658 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:24:33.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T16:24:33.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:33.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:34.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:35.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:35.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:35.061 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:24:35.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T16:24:35.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:35.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:35.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:36.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:36.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:36.386 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:24:36.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T16:24:36.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:36.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:36.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:37.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:37.789 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:37.789 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:24:37.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T16:24:37.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:37.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:38.306 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:39.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:39.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:39.309 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:24:39.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T16:24:39.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:39.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:40.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:41.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:41.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:41.001 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:24:41.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T16:24:41.001 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:41.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:41.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:42.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:42.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:42.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T16:24:42.623 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:24:42.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:42.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:43.475 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:44.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:44.476 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:44.477 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:24:44.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T16:24:44.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:44.477 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:44.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:45.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:45.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:45.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T16:24:45.951 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:24:45.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:45.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:46.639 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:47.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:47.641 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:47.641 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:24:47.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T16:24:47.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:47.642 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:48.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:49.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:49.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:49.356 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:24:49.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T16:24:49.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:49.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:50.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:51.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:51.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:51.216 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:24:51.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T16:24:51.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:51.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:51.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:24:52.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:24:52.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:24:52.573 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T16:24:52.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T16:24:52.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:24:52.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:24:52.920 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2162387661,v1:127.0.0.1:6819/2162387661] [v2:127.0.0.1:6820/2162387661,v1:127.0.0.1:6821/2162387661] exists,up 9ef8cbfa-82b4-4132-af1e-40a5fb7744c2 2024-09-13T16:24:52.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:24:52.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:24:52.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:24:52.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:442: TEST_scrub_permit_time: create_pool test 1 1 2024-09-13T16:24:52.921 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-13T16:24:53.434 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'test' created 2024-09-13T16:24:53.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T16:24:54.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:443: TEST_scrub_permit_time: wait_for_clean 2024-09-13T16:24:54.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:24:54.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:24:54.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:24:54.462 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:24:54.463 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:24:54.463 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:24:54.464 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:24:54.464 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:24:54.464 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:24:54.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:24:54.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:24:54.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:24:54.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:24:54.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:24:54.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:24:55.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:24:55.021 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:24:55.021 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:24:55.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:24:55.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:24:55.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:24:55.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-13T16:24:55.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-13T16:24:55.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-13T16:24:55.162 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:24:55.163 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:24:55.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-13T16:24:55.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-13T16:24:55.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672974' 2024-09-13T16:24:55.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:24:55.443 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:24:55.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-13T16:24:55.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-13T16:24:55.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672974 2-64424509448' 2024-09-13T16:24:55.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:24:55.591 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-13T16:24:55.591 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:24:55.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:24:55.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-13T16:24:55.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:24:55.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-13T16:24:55.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-13T16:24:55.594 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836504 2024-09-13T16:24:55.594 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:24:55.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-13T16:24:55.924 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:24:55.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:24:55.925 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-13T16:24:55.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:24:55.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-13T16:24:55.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:24:55.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-13T16:24:55.929 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672974 2024-09-13T16:24:55.929 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-13T16:24:55.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:24:56.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672973 -lt 42949672974 2024-09-13T16:24:56.255 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:24:57.257 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:24:57.257 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:24:57.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672974 2024-09-13T16:24:57.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:24:57.570 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-13T16:24:57.570 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:24:57.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:24:57.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-13T16:24:57.573 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:24:57.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-13T16:24:57.574 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509448 2024-09-13T16:24:57.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-13T16:24:57.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:24:57.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-13T16:24:57.897 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:24:57.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:24:57.898 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:24:58.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:24:58.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:24:58.269 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:24:58.269 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:24:58.269 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:24:58.269 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:24:58.269 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:24:58.270 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:24:58.568 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:24:58.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:24:58.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:24:58.568 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:24:58.948 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:24:58.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:24:58.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:24:58.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:446: TEST_scrub_permit_time: get_pg test SOMETHING 2024-09-13T16:24:58.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-13T16:24:58.949 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-13T16:24:58.950 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-13T16:24:58.950 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test SOMETHING 2024-09-13T16:24:59.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:446: TEST_scrub_permit_time: local pgid=1.0 2024-09-13T16:24:59.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:447: TEST_scrub_permit_time: get_primary test SOMETHING 2024-09-13T16:24:59.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-13T16:24:59.317 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-13T16:24:59.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test SOMETHING 2024-09-13T16:24:59.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T16:24:59.640 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:447: TEST_scrub_permit_time: local primary=1 2024-09-13T16:24:59.640 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:448: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:24:59.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:24:59.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:24:59.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:24:59.641 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:24:59.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:448: TEST_scrub_permit_time: local last_scrub=2024-09-13T16:24:53.294702+0000 2024-09-13T16:24:59.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: ceph tell 1.0 schedule-scrub 86400 2024-09-13T16:25:00.131 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:25:00.132 INFO:tasks.workunit.client.0.smithi112.stdout: "deep": false, 2024-09-13T16:25:00.132 INFO:tasks.workunit.client.0.smithi112.stdout: "must": false, 2024-09-13T16:25:00.132 INFO:tasks.workunit.client.0.smithi112.stdout: "stamp": "2024-09-12T16:23:20.134142+0000" 2024-09-13T16:25:00.132 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:25:00.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i=0 )) 2024-09-13T16:25:00.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:00.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:00.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:00.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:00.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:00.147 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:00.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-13T16:24:53.294702+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:00.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:01.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:01.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:01.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:01.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:01.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:01.453 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:01.454 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:01.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:01.762 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:02.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:02.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:02.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:02.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:02.764 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:02.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:02.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:03.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:03.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:04.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:04.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:04.077 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:04.077 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:04.077 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:04.077 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:04.077 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:04.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:04.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:05.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:05.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:05.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:05.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:05.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:05.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:05.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:05.735 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:05.736 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:06.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:06.737 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:06.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:06.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:06.738 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:06.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:06.739 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:07.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:07.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:08.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:08.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:08.058 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:08.058 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:08.058 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:08.059 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:08.059 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:08.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:08.366 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:09.367 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:09.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:09.368 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:09.368 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:09.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:09.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:09.369 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:09.702 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:09.703 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:10.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:10.704 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:10.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:10.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:10.705 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:10.706 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:10.706 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:11.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:11.064 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:12.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:12.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:12.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:12.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:12.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:12.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:12.067 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:12.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:12.397 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:13.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:13.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:13.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:13.399 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:13.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:13.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:13.400 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:13.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:13.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:14.728 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:14.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:14.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:14.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:14.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:14.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:14.731 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:15.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:15.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:16.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:16.067 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:16.068 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:16.068 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:16.068 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:16.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:16.069 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:16.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:16.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:17.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:17.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:17.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:17.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:17.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:17.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:17.394 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:17.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:17.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:18.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:18.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:18.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:18.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:18.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:18.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:18.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:19.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:19.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:20.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:20.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:20.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:20.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:20.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:20.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:20.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:20.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:20.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:21.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:21.386 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:21.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:21.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:21.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:21.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:21.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:21.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:21.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:22.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:22.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:22.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:22.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:22.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:22.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:22.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:23.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:23.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:24.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:24.063 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:24.064 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:24.064 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:24.064 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:24.064 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:24.064 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:25.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:25.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:25.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:25.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:25.392 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:25.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:25.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:25.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:25.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:26.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:26.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:26.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:26.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:26.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:26.722 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:26.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:27.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:27.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:28.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:28.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:28.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:28.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:28.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:28.076 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:28.077 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:28.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:28.403 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:29.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:29.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:29.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:29.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:29.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:29.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:29.406 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:29.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:29.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:30.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:30.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:30.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:30.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:30.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:30.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:30.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:31.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:32.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:32.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:32.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:32.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:32.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:32.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:32.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:32.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:32.370 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:33.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:34.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:34.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:34.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:34.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:34.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:34.056 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:34.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:34.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:34.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:34.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:34.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:34.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:34.697 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:34.697 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:34.697 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:35.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:35.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:36.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:36.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:36.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:36.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:36.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:36.063 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:36.064 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:36.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:36.378 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:37.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:37.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:37.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:37.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:37.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:37.381 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:37.381 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:37.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:37.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:38.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:38.693 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:38.694 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2024-09-13T16:25:38.694 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:25:38.694 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:25:38.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:25:38.695 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:25:39.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:457: TEST_scrub_permit_time: test 2024-09-12T16:23:20.134142+0000 '>' 2024-09-13T16:24:53.294702+0000 2024-09-13T16:25:39.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:460: TEST_scrub_permit_time: sleep 1 2024-09-13T16:25:40.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i++ )) 2024-09-13T16:25:40.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:456: TEST_scrub_permit_time: (( i < 30 )) 2024-09-13T16:25:40.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:462: TEST_scrub_permit_time: perf_counters td/osd-scrub-test 3 2024-09-13T16:25:40.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-09-13T16:25:40.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2024-09-13T16:25:40.012 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2024-09-13T16:25:40.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2024-09-13T16:25:40.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:25:40.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-09-13T16:25:40.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:25:40.154 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:25:40.154 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:25:40.154 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:25:40.154 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:25:40.154 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:25:40.154 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:25:40.154 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.154 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:25:40.155 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:25:40.155 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:25:40.155 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:25:40.155 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:25:40.155 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.155 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.155 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.155 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.155 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:25:40.155 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:25:40.155 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.155 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.155 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.156 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.156 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:25:40.156 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:25:40.156 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:25:40.156 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:25:40.156 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:25:40.156 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:25:40.156 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:25:40.156 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.156 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.156 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.156 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.157 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:25:40.157 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:25:40.157 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:25:40.157 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:25:40.157 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.157 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.157 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.157 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.157 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:25:40.157 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.157 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.157 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:25:40.157 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:25:40.158 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:25:40.158 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:25:40.158 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:25:40.158 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:25:40.158 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.158 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:25:40.158 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:25:40.158 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:25:40.158 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:25:40.158 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:25:40.158 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.158 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.158 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.159 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.159 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:25:40.159 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:25:40.159 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.159 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.159 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.159 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.159 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:25:40.159 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:25:40.159 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:25:40.159 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:25:40.159 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:25:40.159 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:25:40.160 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:25:40.160 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.160 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.160 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.160 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.160 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:25:40.160 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:25:40.160 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:25:40.160 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:25:40.160 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.160 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.160 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.160 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.161 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:25:40.161 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.161 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.161 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:25:40.161 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:25:40.161 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:25:40.161 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:25:40.161 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:25:40.161 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:25:40.161 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.161 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:25:40.161 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:25:40.161 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:25:40.162 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:25:40.162 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:25:40.162 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.162 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.162 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.162 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.162 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:25:40.162 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:25:40.162 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.162 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.162 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.162 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.162 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:25:40.163 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:25:40.163 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:25:40.163 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:25:40.163 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:25:40.163 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:25:40.163 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:25:40.163 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.163 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.163 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.163 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.163 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:25:40.163 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:25:40.163 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:25:40.164 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:25:40.164 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.164 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.164 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.164 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.164 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:25:40.164 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.164 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.164 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:25:40.164 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:25:40.164 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:25:40.164 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:25:40.164 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:25:40.164 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:25:40.165 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.165 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:25:40.165 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:25:40.165 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:25:40.165 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:25:40.165 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:25:40.165 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.165 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.165 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.165 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.165 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:25:40.165 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:25:40.165 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.165 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.166 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.166 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.166 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:25:40.166 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:25:40.166 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:25:40.166 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:25:40.166 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:25:40.166 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:25:40.166 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:25:40.167 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.167 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.167 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.167 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.167 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:25:40.167 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:25:40.167 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:25:40.167 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:25:40.167 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.167 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.167 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.167 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.168 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:25:40.168 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.168 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.168 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:25:40.168 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:25:40.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:25:40.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-09-13T16:25:40.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:25:40.290 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:25:40.290 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:25:40.291 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:25:40.291 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:25:40.291 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:25:40.291 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:25:40.291 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.291 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:25:40.291 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:25:40.291 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:25:40.291 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:25:40.291 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:25:40.291 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.291 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.291 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.292 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.292 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:25:40.292 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:25:40.292 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.292 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.292 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.292 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.292 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:25:40.292 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:25:40.292 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:25:40.292 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:25:40.292 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:25:40.292 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:25:40.293 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:25:40.293 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.293 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.293 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.293 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.293 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:25:40.293 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:25:40.293 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:25:40.293 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:25:40.293 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.293 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.293 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.293 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.294 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:25:40.294 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.294 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.294 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:25:40.294 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:25:40.294 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:25:40.294 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:25:40.294 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:25:40.294 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:25:40.294 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.294 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:25:40.294 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:25:40.294 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:25:40.294 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:25:40.295 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:25:40.295 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.295 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.295 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.295 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.295 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:25:40.295 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:25:40.295 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.295 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.295 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.295 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.295 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:25:40.295 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:25:40.295 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:25:40.296 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:25:40.296 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:25:40.296 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:25:40.296 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:25:40.296 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.296 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.296 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.296 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.296 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:25:40.296 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:25:40.296 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:25:40.296 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:25:40.296 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.297 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.297 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.297 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.297 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:25:40.297 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.297 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.297 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:25:40.298 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:25:40.298 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:25:40.298 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:25:40.298 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:25:40.298 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:25:40.298 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.298 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:25:40.298 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:25:40.298 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:25:40.298 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:25:40.298 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:25:40.298 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.299 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.299 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.299 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.299 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:25:40.299 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:25:40.299 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.299 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.299 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.299 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.299 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:25:40.299 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:25:40.300 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:25:40.300 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:25:40.300 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:25:40.300 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:25:40.300 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:25:40.300 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.300 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.300 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.300 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.301 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:25:40.301 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:25:40.301 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:25:40.301 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:25:40.301 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.301 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.301 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.301 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.301 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:25:40.301 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.302 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.302 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:25:40.302 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:25:40.302 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:25:40.302 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:25:40.302 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:25:40.302 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:25:40.302 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.302 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:25:40.302 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:25:40.303 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:25:40.303 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:25:40.303 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:25:40.303 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.303 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.303 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.303 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.303 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:25:40.303 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:25:40.303 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.303 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.304 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.304 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.304 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:25:40.304 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:25:40.304 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:25:40.304 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:25:40.304 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:25:40.304 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:25:40.304 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:25:40.305 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.305 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.305 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.305 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.305 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:25:40.305 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:25:40.305 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:25:40.305 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:25:40.306 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.306 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.306 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.306 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.306 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:25:40.306 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.306 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.306 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:25:40.306 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:25:40.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:25:40.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2024-09-13T16:25:40.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:25:40.433 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:25:40.433 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:25:40.433 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:25:40.433 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:25:40.433 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:25:40.433 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:25:40.433 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.433 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:25:40.433 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:25:40.433 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:25:40.433 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:25:40.433 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:25:40.434 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.434 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.434 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.434 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.434 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:25:40.434 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:25:40.434 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.434 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.434 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.434 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.434 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:25:40.434 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:25:40.434 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:25:40.434 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:25:40.435 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:25:40.435 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:25:40.435 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:25:40.435 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.435 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.435 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.435 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.435 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:25:40.435 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:25:40.435 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:25:40.435 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:25:40.435 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.435 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.436 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.436 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.436 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:25:40.436 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.436 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.436 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:25:40.436 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:25:40.436 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:25:40.436 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:25:40.436 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:25:40.436 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:25:40.436 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.436 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:25:40.436 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:25:40.437 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:25:40.437 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:25:40.437 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:25:40.437 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.437 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.437 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.437 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.437 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:25:40.437 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:25:40.437 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.437 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.437 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.437 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.438 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:25:40.438 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:25:40.438 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:25:40.438 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:25:40.438 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:25:40.438 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:25:40.438 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:25:40.438 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.438 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.438 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.438 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.438 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:25:40.438 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:25:40.439 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:25:40.439 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:25:40.439 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.439 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.439 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.439 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.439 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:25:40.439 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.439 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.439 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:25:40.439 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:25:40.439 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:25:40.439 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:25:40.439 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:25:40.440 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:25:40.440 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.440 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:25:40.440 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:25:40.440 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:25:40.440 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:25:40.440 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:25:40.440 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.440 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.440 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.440 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.440 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:25:40.440 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:25:40.441 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.441 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.441 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.441 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.441 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:25:40.441 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:25:40.441 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:25:40.441 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:25:40.441 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:25:40.441 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:25:40.441 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:25:40.441 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.441 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.442 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.442 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.442 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:25:40.442 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:25:40.442 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:25:40.442 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:25:40.442 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.442 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.442 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.442 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.442 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:25:40.442 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.442 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.442 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:25:40.443 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:25:40.443 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:25:40.443 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:25:40.443 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:25:40.443 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:25:40.443 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.443 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:25:40.443 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:25:40.443 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:25:40.443 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:25:40.443 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:25:40.443 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.443 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.444 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.444 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.444 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:25:40.444 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:25:40.444 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.444 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.444 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.444 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.444 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:25:40.444 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:25:40.444 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:25:40.444 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:25:40.444 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:25:40.444 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:25:40.445 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:25:40.445 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.445 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.445 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.445 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.445 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:25:40.445 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:25:40.445 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:25:40.446 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:25:40.446 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:25:40.446 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:25:40.446 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:25:40.446 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:25:40.446 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:25:40.446 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.446 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:25:40.446 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:25:40.446 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:25:40.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_scrub_permit_time ------------------' 2024-09-13T16:25:40.447 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Teardown Test TEST_scrub_permit_time ------------------ 2024-09-13T16:25:40.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-09-13T16:25:40.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:25:40.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:25:40.448 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:25:40.448 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:25:40.448 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:25:40.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:25:40.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:25:40.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:25:40.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:25:40.555 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:25:40.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:25:40.555 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:25:40.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:25:40.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:25:40.557 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:25:40.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:25:40.557 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:25:40.558 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:25:40.558 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:25:40.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:25:40.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:25:40.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:25:40.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:25:40.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:25:40.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:25:40.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:25:40.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:25:40.577 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:25:40.578 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Complete Test TEST_scrub_permit_time ------------------ 2024-09-13T16:25:40.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_scrub_permit_time ------------------' 2024-09-13T16:25:40.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-09-13T16:25:40.578 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Prepare Test TEST_scrub_test ------------------- 2024-09-13T16:25:40.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_scrub_test -------------------' 2024-09-13T16:25:40.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-09-13T16:25:40.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2024-09-13T16:25:40.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2024-09-13T16:25:40.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:25:40.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:25:40.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:25:40.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:25:40.580 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:25:40.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:25:40.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:25:40.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:25:40.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:25:40.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:25:40.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:25:40.582 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:25:40.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:25:40.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:25:40.583 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:25:40.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:25:40.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:25:40.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:25:40.584 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:25:40.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:25:40.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:25:40.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:25:40.587 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:25:40.587 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:25:40.587 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:25:40.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:25:40.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:25:40.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:25:40.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2024-09-13T16:25:40.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T16:25:40.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:25:40.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:25:40.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.528094 2024-09-13T16:25:40.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T16:25:40.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-13T16:25:40.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T16:25:40.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-09-13T16:25:40.591 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Run Test TEST_scrub_test ----------------------- 2024-09-13T16:25:40.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_scrub_test -----------------------' 2024-09-13T16:25:40.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_scrub_test td/osd-scrub-test 2024-09-13T16:25:40.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:52: TEST_scrub_test: local dir=td/osd-scrub-test 2024-09-13T16:25:40.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:53: TEST_scrub_test: local poolname=test 2024-09-13T16:25:40.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:54: TEST_scrub_test: local OSDS=3 2024-09-13T16:25:40.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:55: TEST_scrub_test: local objects=15 2024-09-13T16:25:40.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:57: TEST_scrub_test: TESTDATA=testdata.528094 2024-09-13T16:25:40.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-09-13T16:25:40.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2024-09-13T16:25:40.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T16:25:40.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T16:25:40.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T16:25:40.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2024-09-13T16:25:40.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-09-13T16:25:41.120 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T16:25:41.120 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:25:41.120 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:25:41.120 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:25:41.121 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:25:41.121 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:25:41.121 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:25:41.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-13T16:25:41.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T16:25:41.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T16:25:41.375 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T16:25:41.375 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T16:25:41.375 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T16:25:41.375 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T16:25:41.375 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T16:25:41.376 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T16:25:41.376 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T16:25:41.376 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:25:41.376 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:25:41.376 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:25:41.377 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-mon.a.asok 2024-09-13T16:25:41.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T16:25:41.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.528094/ceph-mon.a.asok config get fsid 2024-09-13T16:25:41.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T16:25:41.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T16:25:41.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T16:25:41.491 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T16:25:41.492 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T16:25:41.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T16:25:41.493 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T16:25:41.493 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T16:25:41.493 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:25:41.493 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:25:41.493 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:25:41.493 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.528094/ceph-mon.a.asok 2024-09-13T16:25:41.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T16:25:41.493 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.528094/ceph-mon.a.asok config get mon_host 2024-09-13T16:25:41.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:60: TEST_scrub_test: run_mgr td/osd-scrub-test x 2024-09-13T16:25:41.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2024-09-13T16:25:41.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T16:25:41.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T16:25:41.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T16:25:41.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2024-09-13T16:25:41.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T16:25:42.481 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T16:25:42.482 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:25:42.482 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:25:42.482 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:25:42.482 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:25:42.482 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:25:42.482 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:25:42.483 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T16:25:42.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T16:25:42.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-09-13T16:25:42.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: ceph_osd_args+='--osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 ' 2024-09-13T16:25:42.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:63: TEST_scrub_test: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-09-13T16:25:42.519 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:64: TEST_scrub_test: expr 3 - 1 2024-09-13T16:25:42.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:64: TEST_scrub_test: seq 0 2 2024-09-13T16:25:42.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:64: TEST_scrub_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:25:42.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:66: TEST_scrub_test: run_osd td/osd-scrub-test 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T16:25:42.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:25:42.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:25:42.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T16:25:42.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:25:42.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2024-09-13T16:25:42.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:25:42.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:25:42.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:25:42.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:25:42.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-09-13T16:25:42.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-09-13T16:25:42.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:25:42.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:25:42.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:25:42.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:25:42.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:25:42.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:25:42.525 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:25:42.525 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:25:42.525 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:25:42.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:25:42.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:25:42.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:25:42.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:25:42.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:25:42.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:25:42.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:25:42.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:25:42.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:25:42.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:25:42.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:25:42.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:25:42.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-09-13T16:25:42.526 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2024-09-13T16:25:42.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:25:42.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=38602c76-559e-4292-8989-9cbf206decee 2024-09-13T16:25:42.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 38602c76-559e-4292-8989-9cbf206decee' 2024-09-13T16:25:42.527 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 38602c76-559e-4292-8989-9cbf206decee 2024-09-13T16:25:42.527 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:25:42.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCGZ+RmI/lbIBAAJFQg8fJeCXO7kTeMyKPTaw== 2024-09-13T16:25:42.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCGZ+RmI/lbIBAAJFQg8fJeCXO7kTeMyKPTaw=="}' 2024-09-13T16:25:42.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 38602c76-559e-4292-8989-9cbf206decee -i td/osd-scrub-test/0/new.json 2024-09-13T16:25:42.851 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:25:42.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2024-09-13T16:25:42.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCGZ+RmI/lbIBAAJFQg8fJeCXO7kTeMyKPTaw== --osd-uuid 38602c76-559e-4292-8989-9cbf206decee 2024-09-13T16:25:42.905 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:25:42.904+0000 7f378a8387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:25:42.944 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:25:42.944+0000 7f378a8387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:25:42.945 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:25:42.944+0000 7f378a8387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:25:42.946 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:25:42.948+0000 7f378a8387c0 -1 bdev(0x55a616fc5c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2024-09-13T16:25:42.947 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:25:42.948+0000 7f378a8387c0 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-09-13T16:25:51.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-09-13T16:25:51.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:25:51.075 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T16:25:51.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T16:25:51.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:25:51.464 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T16:25:51.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T16:25:51.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T16:25:51.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:25:51.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:25:51.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:25:51.498 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:25:51.499+0000 7faaca6e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:25:51.562 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:25:51.563+0000 7faaca6e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:25:51.565 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:25:51.563+0000 7faaca6e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:25:51.785 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T16:25:51.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:25:51.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T16:25:51.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:25:51.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:25:51.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:25:51.786 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:25:51.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:25:51.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:25:51.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:25:52.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:25:53.065 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:25:53.063+0000 7faaca6e97c0 -1 Falling back to public interface 2024-09-13T16:25:53.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:25:53.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:25:53.098 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:25:53.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:25:53.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:25:53.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:25:53.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:25:54.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:25:54.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:25:54.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:25:54.404 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:25:54.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:25:54.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:25:54.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:25:55.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:25:55.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:25:55.731 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:25:55.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:25:55.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:25:55.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:25:56.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:25:56.051 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:25:56.051+0000 7faaca6e97c0 -1 osd.0 0 log_to_monitors true 2024-09-13T16:25:57.043 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:25:57.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:25:57.044 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:25:57.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:25:57.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:25:57.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:25:57.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:25:58.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:25:58.512 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:25:58.513 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:25:58.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:25:58.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:25:58.513 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:25:58.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:25:59.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:25:59.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:25:59.842 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:25:59.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T16:25:59.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:25:59.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:00.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:01.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:01.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:01.266 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:26:01.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T16:26:01.266 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:01.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:01.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:02.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:02.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:02.581 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:26:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T16:26:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:02.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:02.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:03.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:03.974 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:03.975 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:26:03.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T16:26:03.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:03.975 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:04.429 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:05.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:05.431 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:05.431 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:26:05.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T16:26:05.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:05.432 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:05.761 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:06.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:06.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:06.764 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:26:06.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T16:26:06.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:06.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:07.273 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:08.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:08.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:08.275 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:26:08.275 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T16:26:08.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:08.276 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:08.609 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:09.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:09.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:09.612 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:26:09.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T16:26:09.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:09.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:10.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:11.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:11.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:11.131 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:26:11.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T16:26:11.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:11.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:11.459 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:12.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:12.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:12.461 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:26:12.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T16:26:12.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:12.462 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:12.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:13.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:13.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:13.779 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:26:13.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T16:26:13.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:13.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:14.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:15.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:15.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:15.263 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:26:15.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T16:26:15.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:15.263 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:16.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:17.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:17.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:17.406 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:26:17.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T16:26:17.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:17.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:18.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:19.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:19.457 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:19.458 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:26:19.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T16:26:19.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:19.458 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:20.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:21.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:21.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:21.390 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:26:21.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T16:26:21.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:21.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:22.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:23.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:23.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:23.383 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:26:23.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T16:26:23.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:23.383 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:23.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:24.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:24.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:24.720 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:26:24.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T16:26:24.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:24.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:25.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:26.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:26.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:26.374 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:26:26.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T16:26:26.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:26.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:27.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:28.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:28.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:28.035 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:26:28.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T16:26:28.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:28.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:26:28.369 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2985513432,v1:127.0.0.1:6803/2985513432] [v2:127.0.0.1:6804/2985513432,v1:127.0.0.1:6805/2985513432] exists,up 38602c76-559e-4292-8989-9cbf206decee 2024-09-13T16:26:28.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:26:28.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:26:28.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:26:28.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:64: TEST_scrub_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:26:28.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:66: TEST_scrub_test: run_osd td/osd-scrub-test 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T16:26:28.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:26:28.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:26:28.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T16:26:28.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:26:28.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2024-09-13T16:26:28.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:26:28.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:26:28.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:26:28.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:26:28.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-09-13T16:26:28.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-09-13T16:26:28.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:26:28.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:26:28.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:26:28.373 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:26:28.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:26:28.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:26:28.374 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:26:28.374 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:26:28.374 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:26:28.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:26:28.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:26:28.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:26:28.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:26:28.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:26:28.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:26:28.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:26:28.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:26:28.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:26:28.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:26:28.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:26:28.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:26:28.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-09-13T16:26:28.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2024-09-13T16:26:28.375 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:26:28.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e2ab7fcd-86a4-4452-ab21-309ce5b1fe26 2024-09-13T16:26:28.376 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 e2ab7fcd-86a4-4452-ab21-309ce5b1fe26 2024-09-13T16:26:28.376 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e2ab7fcd-86a4-4452-ab21-309ce5b1fe26' 2024-09-13T16:26:28.376 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:26:28.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC0Z+RmMRRlFxAA2wkbg0JRogBpg1jTLB6q2A== 2024-09-13T16:26:28.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC0Z+RmMRRlFxAA2wkbg0JRogBpg1jTLB6q2A=="}' 2024-09-13T16:26:28.391 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e2ab7fcd-86a4-4452-ab21-309ce5b1fe26 -i td/osd-scrub-test/1/new.json 2024-09-13T16:26:28.819 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:26:28.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2024-09-13T16:26:28.840 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQC0Z+RmMRRlFxAA2wkbg0JRogBpg1jTLB6q2A== --osd-uuid e2ab7fcd-86a4-4452-ab21-309ce5b1fe26 2024-09-13T16:26:28.878 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:26:28.878+0000 7f815e6f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:26:28.962 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:26:28.962+0000 7f815e6f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:26:28.964 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:26:28.962+0000 7f815e6f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:26:28.965 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:26:28.962+0000 7f815e6f97c0 -1 bdev(0x563065260800 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2024-09-13T16:26:28.965 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:26:28.962+0000 7f815e6f97c0 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-09-13T16:26:37.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-09-13T16:26:37.763 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:26:37.764 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T16:26:37.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T16:26:37.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:26:38.172 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T16:26:38.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T16:26:38.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T16:26:38.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:26:38.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:26:38.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:26:38.206 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:26:38.206+0000 7f5800d927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:26:38.251 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:26:38.250+0000 7f5800d927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:26:38.253 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:26:38.250+0000 7f5800d927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:26:38.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T16:26:38.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:26:38.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T16:26:38.493 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:26:38.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:26:38.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:38.494 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:26:38.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:26:38.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:38.494 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:38.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:39.753 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:26:39.750+0000 7f5800d927c0 -1 Falling back to public interface 2024-09-13T16:26:39.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:39.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:39.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:26:39.805 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:26:39.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:39.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:40.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:41.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:41.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:41.123 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:26:41.124 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:26:41.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:41.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:41.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:42.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:42.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:42.436 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:26:42.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:26:42.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:42.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:42.724 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:26:42.722+0000 7f5800d927c0 -1 osd.1 0 log_to_monitors true 2024-09-13T16:26:42.766 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:43.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:43.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:43.768 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:26:43.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:26:43.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:43.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:44.100 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:45.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:45.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:45.102 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:26:45.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:26:45.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:45.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:45.433 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:46.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:46.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:46.435 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:26:46.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T16:26:46.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:46.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:46.925 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:47.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:47.927 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:47.927 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:26:47.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T16:26:47.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:47.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:48.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:49.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:49.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:49.260 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:26:49.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T16:26:49.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:49.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:49.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:50.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:50.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:50.593 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:26:50.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T16:26:50.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:50.593 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:50.932 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:51.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:51.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:51.934 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:26:51.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T16:26:51.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:51.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:52.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:53.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:53.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:53.264 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:26:53.264 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T16:26:53.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:53.265 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:53.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:54.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:54.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:54.659 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:26:54.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T16:26:54.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:54.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:55.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:56.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:56.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:56.136 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:26:56.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T16:26:56.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:56.137 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:56.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:57.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:57.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:57.587 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:26:57.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T16:26:57.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:57.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:58.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:26:59.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:26:59.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:26:59.057 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:26:59.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T16:26:59.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:26:59.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:26:59.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:00.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:00.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:00.393 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:27:00.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T16:27:00.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:00.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:27:00.916 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:01.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:01.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:01.919 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:27:01.919 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T16:27:01.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:01.920 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:27:02.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:03.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:03.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:03.248 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:27:03.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T16:27:03.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:03.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:27:03.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:04.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:04.621 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:04.621 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:27:04.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T16:27:04.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:04.622 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:27:05.310 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:06.312 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:06.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:06.313 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:27:06.313 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T16:27:06.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:06.314 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:27:06.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:07.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:07.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:07.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T16:27:07.867 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:27:07.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:07.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:27:08.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:09.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:09.913 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:09.913 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:27:09.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T16:27:09.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:09.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:27:10.644 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:11.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:11.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:11.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T16:27:11.646 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:27:11.646 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:11.647 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:27:12.280 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:13.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:13.282 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:13.282 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:27:13.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T16:27:13.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:13.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:27:13.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:14.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:14.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:14.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T16:27:14.718 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T16:27:14.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:14.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:27:15.056 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:16.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:16.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:16.058 INFO:tasks.workunit.client.0.smithi112.stdout:26 2024-09-13T16:27:16.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 26 2024-09-13T16:27:16.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:16.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:27:16.443 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2664681832,v1:127.0.0.1:6811/2664681832] [v2:127.0.0.1:6812/2664681832,v1:127.0.0.1:6813/2664681832] exists,up e2ab7fcd-86a4-4452-ab21-309ce5b1fe26 2024-09-13T16:27:16.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:27:16.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:27:16.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:27:16.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:64: TEST_scrub_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:27:16.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:66: TEST_scrub_test: run_osd td/osd-scrub-test 2 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T16:27:16.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2024-09-13T16:27:16.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:27:16.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T16:27:16.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:27:16.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2024-09-13T16:27:16.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:27:16.446 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:27:16.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:27:16.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:27:16.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-13T16:27:16.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-13T16:27:16.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:27:16.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:27:16.447 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:27:16.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:27:16.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:27:16.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:27:16.447 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:27:16.447 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:27:16.447 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:27:16.448 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:27:16.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:27:16.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:27:16.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:27:16.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:27:16.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:27:16.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:27:16.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:27:16.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:27:16.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:27:16.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:27:16.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:27:16.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-09-13T16:27:16.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2024-09-13T16:27:16.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:27:16.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8ce30b46-ace2-490e-9845-7cade58e98dc 2024-09-13T16:27:16.450 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 8ce30b46-ace2-490e-9845-7cade58e98dc 2024-09-13T16:27:16.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 8ce30b46-ace2-490e-9845-7cade58e98dc' 2024-09-13T16:27:16.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:27:16.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDkZ+RmNdb6GxAA/RD7uXG7rxlR0PUKe65xqg== 2024-09-13T16:27:16.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDkZ+RmNdb6GxAA/RD7uXG7rxlR0PUKe65xqg=="}' 2024-09-13T16:27:16.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8ce30b46-ace2-490e-9845-7cade58e98dc -i td/osd-scrub-test/2/new.json 2024-09-13T16:27:17.077 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:27:17.101 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2024-09-13T16:27:17.102 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDkZ+RmNdb6GxAA/RD7uXG7rxlR0PUKe65xqg== --osd-uuid 8ce30b46-ace2-490e-9845-7cade58e98dc 2024-09-13T16:27:17.135 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:27:17.133+0000 7fa0097627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:27:17.213 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:27:17.209+0000 7fa0097627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:27:17.215 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:27:17.213+0000 7fa0097627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:27:17.215 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:27:17.213+0000 7fa0097627c0 -1 bdev(0x55c6cd826800 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2024-09-13T16:27:17.216 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:27:17.213+0000 7fa0097627c0 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-09-13T16:27:25.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-09-13T16:27:25.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:27:25.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T16:27:25.719 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T16:27:25.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:27:26.127 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T16:27:26.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T16:27:26.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T16:27:26.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:27:26.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:27:26.128 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:27:26.161 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:27:26.157+0000 7f761a1d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:27:26.202 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:27:26.201+0000 7f761a1d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:27:26.204 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:27:26.201+0000 7f761a1d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:27:26.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T16:27:26.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:27:26.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T16:27:26.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:27:26.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:27:26.434 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:26.435 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:27:26.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:27:26.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:26.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:26.740 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:27.248 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:27:27.245+0000 7f761a1d47c0 -1 Falling back to public interface 2024-09-13T16:27:27.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:27.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:27.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:27:27.742 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:27:27.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:27.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:28.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:29.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:27:29.059 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:27:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:29.060 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:29.368 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:30.369 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:30.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:30.585 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:27:30.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:27:30.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:30.586 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:30.586 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:27:30.385+0000 7f761a1d47c0 -1 osd.2 0 log_to_monitors true 2024-09-13T16:27:30.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:31.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:31.711 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:31.711 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:27:31.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:27:31.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:31.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:32.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:33.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:33.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:33.210 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:27:33.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:27:33.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:33.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:33.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:34.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:34.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:34.716 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:27:34.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T16:27:34.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:34.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:35.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:36.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:36.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:36.053 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:27:36.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T16:27:36.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:36.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:36.602 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:37.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:37.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:37.604 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:27:37.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T16:27:37.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:37.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:37.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:38.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:38.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:38.930 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T16:27:38.930 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:27:38.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:38.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:39.246 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:40.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:40.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:40.248 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:27:40.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T16:27:40.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:40.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:40.659 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:41.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:41.661 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:41.661 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:27:41.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T16:27:41.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:41.662 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:42.182 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:43.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:43.184 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:43.185 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:27:43.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T16:27:43.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:43.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:43.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:44.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:44.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:44.731 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:27:44.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T16:27:44.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:44.732 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:45.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:46.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:46.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:46.058 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:27:46.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T16:27:46.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:46.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:46.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:47.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:47.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:47.723 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:27:47.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T16:27:47.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:47.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:48.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:49.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:49.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:49.220 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:27:49.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T16:27:49.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:49.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:49.718 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:50.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:50.720 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:50.721 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:27:50.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T16:27:50.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:50.721 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:51.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:52.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:52.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:52.362 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:27:52.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T16:27:52.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:52.362 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:53.115 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:54.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:54.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:54.118 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:27:54.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T16:27:54.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:54.118 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:54.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:55.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:55.781 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:27:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T16:27:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:55.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:56.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:57.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:57.111 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:57.111 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:27:57.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T16:27:57.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:57.112 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:57.435 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:58.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:58.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:58.437 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:27:58.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T16:27:58.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:58.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:27:58.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:27:59.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:27:59.945 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:27:59.946 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:27:59.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T16:27:59.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:27:59.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:28:00.979 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:28:01.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:28:01.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:28:01.981 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:28:01.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T16:28:01.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:28:01.982 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:28:02.705 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:28:03.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:28:03.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:28:03.707 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T16:28:03.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T16:28:03.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:28:03.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:28:04.030 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:28:05.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:28:05.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:28:05.032 INFO:tasks.workunit.client.0.smithi112.stdout:26 2024-09-13T16:28:05.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 26 2024-09-13T16:28:05.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:28:05.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:28:05.362 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1741209660,v1:127.0.0.1:6819/1741209660] [v2:127.0.0.1:6820/1741209660,v1:127.0.0.1:6821/1741209660] exists,up 8ce30b46-ace2-490e-9845-7cade58e98dc 2024-09-13T16:28:05.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:28:05.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:28:05.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:28:05.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:70: TEST_scrub_test: create_pool test 1 1 2024-09-13T16:28:05.363 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-13T16:28:05.841 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'test' created 2024-09-13T16:28:05.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-13T16:28:06.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:71: TEST_scrub_test: wait_for_clean 2024-09-13T16:28:06.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:28:06.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:28:06.866 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:28:06.867 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:28:06.868 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:28:06.868 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:28:06.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:28:06.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:28:06.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:28:06.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:28:06.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:28:06.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:28:06.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:28:06.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:28:06.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:28:07.392 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:28:07.392 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:28:07.393 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:28:07.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:28:07.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:28:07.393 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:28:07.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836508 2024-09-13T16:28:07.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836508 2024-09-13T16:28:07.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508' 2024-09-13T16:28:07.575 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:28:07.575 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:28:07.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672980 2024-09-13T16:28:07.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672980 2024-09-13T16:28:07.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672980' 2024-09-13T16:28:07.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:28:07.814 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:28:07.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509447 2024-09-13T16:28:07.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509447 2024-09-13T16:28:07.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672980 2-64424509447' 2024-09-13T16:28:07.958 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:28:07.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836508 2024-09-13T16:28:07.959 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:28:07.959 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:28:07.960 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836508 2024-09-13T16:28:07.960 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:28:07.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836508 2024-09-13T16:28:07.961 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836508 2024-09-13T16:28:07.961 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-13T16:28:07.962 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:28:08.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836508 2024-09-13T16:28:08.283 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:28:09.285 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:28:09.285 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:28:09.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836508 2024-09-13T16:28:09.610 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:28:09.611 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672980 2024-09-13T16:28:09.612 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:28:09.613 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:28:09.613 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672980 2024-09-13T16:28:09.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:28:09.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672980 2024-09-13T16:28:09.615 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672980 2024-09-13T16:28:09.615 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-13T16:28:09.615 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:28:10.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-13T16:28:10.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:28:10.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:28:10.023 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509447 2024-09-13T16:28:10.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:28:10.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509447 2024-09-13T16:28:10.025 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:28:10.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509447 2024-09-13T16:28:10.026 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509447 2024-09-13T16:28:10.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-13T16:28:10.027 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:28:10.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509447 2024-09-13T16:28:10.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:28:10.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:28:10.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:28:10.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:28:10.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:28:10.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:28:10.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:28:10.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:28:10.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:28:10.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:28:10.725 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:28:11.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:28:11.074 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:28:11.075 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:28:11.075 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:28:11.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:28:11.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:28:11.443 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:28:11.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:72: TEST_scrub_test: awk '{ print $2 }' 2024-09-13T16:28:11.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:72: TEST_scrub_test: ceph osd dump 2024-09-13T16:28:11.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:72: TEST_scrub_test: grep '^pool.*['\'']test['\'']' 2024-09-13T16:28:11.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:72: TEST_scrub_test: poolid=1 2024-09-13T16:28:11.768 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:74: TEST_scrub_test: dd if=/dev/urandom of=testdata.528094 bs=1032 count=1 2024-09-13T16:28:11.768 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T16:28:11.769 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T16:28:11.769 INFO:tasks.workunit.client.0.smithi112.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 0.00011519 s, 9.0 MB/s 2024-09-13T16:28:11.769 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: seq 1 15 2024-09-13T16:28:11.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-13T16:28:11.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rados -p test put obj1 testdata.528094 2024-09-13T16:28:12.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-13T16:28:12.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rados -p test put obj2 testdata.528094 2024-09-13T16:28:12.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-13T16:28:12.229 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rados -p test put obj3 testdata.528094 2024-09-13T16:28:12.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-13T16:28:12.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rados -p test put obj4 testdata.528094 2024-09-13T16:28:12.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-13T16:28:12.688 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rados -p test put obj5 testdata.528094 2024-09-13T16:28:12.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-13T16:28:12.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rados -p test put obj6 testdata.528094 2024-09-13T16:28:13.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-13T16:28:13.107 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rados -p test put obj7 testdata.528094 2024-09-13T16:28:13.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-13T16:28:13.307 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rados -p test put obj8 testdata.528094 2024-09-13T16:28:13.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-13T16:28:13.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rados -p test put obj9 testdata.528094 2024-09-13T16:28:13.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-13T16:28:13.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rados -p test put obj10 testdata.528094 2024-09-13T16:28:13.995 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-13T16:28:13.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rados -p test put obj11 testdata.528094 2024-09-13T16:28:14.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-13T16:28:14.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rados -p test put obj12 testdata.528094 2024-09-13T16:28:14.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-13T16:28:14.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rados -p test put obj13 testdata.528094 2024-09-13T16:28:14.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-13T16:28:14.597 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rados -p test put obj14 testdata.528094 2024-09-13T16:28:14.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: for i in `seq 1 $objects` 2024-09-13T16:28:14.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rados -p test put obj15 testdata.528094 2024-09-13T16:28:15.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:79: TEST_scrub_test: rm -f testdata.528094 2024-09-13T16:28:15.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:81: TEST_scrub_test: get_primary test obj1 2024-09-13T16:28:15.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-13T16:28:15.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T16:28:15.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-13T16:28:15.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T16:28:15.346 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:81: TEST_scrub_test: local primary=1 2024-09-13T16:28:15.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:82: TEST_scrub_test: get_not_primary test obj1 2024-09-13T16:28:15.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2024-09-13T16:28:15.346 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-13T16:28:15.347 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2024-09-13T16:28:15.347 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-13T16:28:15.347 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-13T16:28:15.347 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-13T16:28:15.348 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T16:28:15.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-13T16:28:15.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2024-09-13T16:28:15.698 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-13T16:28:16.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:82: TEST_scrub_test: local otherosd=0 2024-09-13T16:28:16.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:83: TEST_scrub_test: '[' 0 = 2 ']' 2024-09-13T16:28:16.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:87: TEST_scrub_test: local anotherosd=2 2024-09-13T16:28:16.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:90: TEST_scrub_test: CORRUPT_DATA=corrupt-data.528094 2024-09-13T16:28:16.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:91: TEST_scrub_test: dd if=/dev/urandom of=corrupt-data.528094 bs=512 count=1 2024-09-13T16:28:16.016 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records in 2024-09-13T16:28:16.016 INFO:tasks.workunit.client.0.smithi112.stderr:1+0 records out 2024-09-13T16:28:16.016 INFO:tasks.workunit.client.0.smithi112.stderr:512 bytes copied, 0.000143367 s, 3.6 MB/s 2024-09-13T16:28:16.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:92: TEST_scrub_test: objectstore_tool td/osd-scrub-test 2 obj1 set-bytes corrupt-data.528094 2024-09-13T16:28:16.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-test 2024-09-13T16:28:16.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T16:28:16.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-13T16:28:16.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T16:28:16.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-test 2 obj1 set-bytes corrupt-data.528094 2024-09-13T16:28:16.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-test 2024-09-13T16:28:16.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T16:28:16.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-13T16:28:16.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T16:28:16.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-test TERM osd.2 2024-09-13T16:28:16.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:28:16.018 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:28:16.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:28:16.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:28:16.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:28:17.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:28:17.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-test 2 obj1 set-bytes corrupt-data.528094 2024-09-13T16:28:17.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-test 2024-09-13T16:28:17.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T16:28:17.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-13T16:28:17.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T16:28:17.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-test/2 2024-09-13T16:28:17.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-test/2 obj1 set-bytes corrupt-data.528094 2024-09-13T16:28:20.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-test 2 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T16:28:20.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-test 2024-09-13T16:28:20.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T16:28:20.045 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T16:28:20.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T16:28:20.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-test/2 2024-09-13T16:28:20.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-09-13T16:28:20.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:28:20.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:28:20.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:28:20.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-09-13T16:28:20.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-09-13T16:28:20.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T16:28:20.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T16:28:20.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-09-13T16:28:20.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T16:28:20.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:28:20.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:28:20.049 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:28:20.049 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:28:20.049 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:28:20.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:28:20.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' 2024-09-13T16:28:20.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:28:20.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-09-13T16:28:20.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-09-13T16:28:20.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:28:20.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:28:20.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:28:20.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:28:20.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T16:28:20.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-09-13T16:28:20.050 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-test/2 2024-09-13T16:28:20.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T16:28:20.051 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.2 2024-09-13T16:28:20.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=93535d72-268c-4aa9-a76d-86d5f6f00591 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.528094/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-09-13T16:28:20.051 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-test/2/whoami 2024-09-13T16:28:20.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T16:28:20.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T16:28:20.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T16:28:20.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T16:28:20.086 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:28:20.088+0000 7f9321d2e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:28:20.090 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:28:20.088+0000 7f9321d2e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:28:20.090 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:28:20.092+0000 7f9321d2e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:28:20.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T16:28:20.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:28:20.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T16:28:20.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:28:20.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:28:20.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:28:20.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:28:20.373 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T16:28:20.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:28:20.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:28:20.692 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:28:21.124 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:28:21.124+0000 7f9321d2e7c0 -1 Falling back to public interface 2024-09-13T16:28:21.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:28:21.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:28:21.694 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:28:21.694 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:28:21.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:28:21.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:28:22.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:28:23.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:28:23.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:28:23.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:28:23.026 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T16:28:23.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:28:23.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:28:23.388 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:28:24.343 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:28:24.343+0000 7f9321d2e7c0 -1 osd.2 19 log_to_monitors true 2024-09-13T16:28:24.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:28:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:28:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:28:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T16:28:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:28:24.390 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:28:24.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:28:25.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:28:25.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2024-09-13T16:28:25.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:28:25.851 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T16:28:25.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:28:25.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:28:26.502 INFO:tasks.workunit.client.0.smithi112.stderr:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 20 last_clean_interval [15,19) [v2:127.0.0.1:6818/1231640029,v1:127.0.0.1:6819/1231640029] [v2:127.0.0.1:6820/1231640029,v1:127.0.0.1:6821/1231640029] exists,up 8ce30b46-ace2-490e-9845-7cade58e98dc 2024-09-13T16:28:26.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:28:26.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:28:26.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:28:26.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T16:28:26.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:28:26.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:28:26.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:28:26.504 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:28:26.504 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:28:26.504 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:28:26.504 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:28:26.504 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:28:26.504 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:28:26.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:28:26.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:28:26.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:28:26.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:28:26.614 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:28:26.614 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:28:26.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:28:26.918 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:28:26.918 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:28:26.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:28:26.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:28:26.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:28:27.061 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836513 2024-09-13T16:28:27.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836513 2024-09-13T16:28:27.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513' 2024-09-13T16:28:27.062 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:28:27.062 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:28:27.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672985 2024-09-13T16:28:27.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672985 2024-09-13T16:28:27.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-42949672985' 2024-09-13T16:28:27.202 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:28:27.202 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:28:27.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=98784247810 2024-09-13T16:28:27.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 98784247810 2024-09-13T16:28:27.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-42949672985 2-98784247810' 2024-09-13T16:28:27.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:28:27.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836513 2024-09-13T16:28:27.405 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:28:27.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:28:27.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836513 2024-09-13T16:28:27.407 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:28:27.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836513 2024-09-13T16:28:27.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-13T16:28:27.408 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836513 2024-09-13T16:28:27.408 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:28:27.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836513 -lt 21474836513 2024-09-13T16:28:27.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:28:27.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672985 2024-09-13T16:28:27.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:28:27.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:28:27.717 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672985 2024-09-13T16:28:27.717 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:28:27.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672985 2024-09-13T16:28:27.719 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-13T16:28:27.719 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 42949672985 2024-09-13T16:28:27.719 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:28:28.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-13T16:28:28.041 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:28:28.042 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-98784247810 2024-09-13T16:28:28.042 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:28:28.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:28:28.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-98784247810 2024-09-13T16:28:28.045 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:28:28.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=98784247810 2024-09-13T16:28:28.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 98784247810' 2024-09-13T16:28:28.046 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 98784247810 2024-09-13T16:28:28.047 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:28:28.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 98784247810 2024-09-13T16:28:28.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:28:29.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:28:29.360 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:28:29.672 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 98784247810 -lt 98784247810 2024-09-13T16:28:29.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:28:29.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:28:29.673 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:28:30.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:28:30.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:28:30.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:28:30.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:28:30.094 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:28:30.095 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:28:30.095 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:28:30.095 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:28:30.387 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:28:30.387 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:28:30.388 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:28:30.388 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:28:30.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:28:30.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:28:30.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:28:30.775 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:93: TEST_scrub_test: rm -f corrupt-data.528094 2024-09-13T16:28:30.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:95: TEST_scrub_test: local pgid=1.0 2024-09-13T16:28:30.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:96: TEST_scrub_test: pg_deep_scrub 1.0 2024-09-13T16:28:30.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=1.0 2024-09-13T16:28:30.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-13T16:28:30.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:28:30.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T16:28:30.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:28:30.777 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-13T16:28:31.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-13T16:28:05.763771+0000 2024-09-13T16:28:31.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 1.0 2024-09-13T16:28:31.362 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2024-09-13T16:28:31.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 1.0 2024-09-13T16:28:05.763771+0000 last_deep_scrub_stamp 2024-09-13T16:28:31.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T16:28:31.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T16:28:05.763771+0000 2024-09-13T16:28:31.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-13T16:28:31.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T16:28:31.379 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:28:31.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-13T16:28:31.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:28:31.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T16:28:31.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:28:31.380 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-13T16:28:31.667 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:28:05.763771+0000 '>' 2024-09-13T16:28:05.763771+0000 2024-09-13T16:28:31.668 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:28:32.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:28:32.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:28:32.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-13T16:28:32.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:28:32.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T16:28:32.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:28:32.671 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-13T16:28:33.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:28:05.763771+0000 '>' 2024-09-13T16:28:05.763771+0000 2024-09-13T16:28:33.068 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:28:34.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:28:34.070 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:28:34.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-13T16:28:34.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:28:34.071 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T16:28:34.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:28:34.072 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-13T16:28:34.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:28:05.763771+0000 '>' 2024-09-13T16:28:05.763771+0000 2024-09-13T16:28:34.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:28:35.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:28:35.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:28:35.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-13T16:28:35.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:28:35.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T16:28:35.395 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:28:35.396 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-13T16:28:35.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:28:05.763771+0000 '>' 2024-09-13T16:28:05.763771+0000 2024-09-13T16:28:35.708 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:28:36.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:28:36.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:28:36.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-13T16:28:36.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:28:36.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T16:28:36.711 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:28:36.711 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-13T16:28:37.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:28:05.763771+0000 '>' 2024-09-13T16:28:05.763771+0000 2024-09-13T16:28:37.040 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:28:38.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:28:38.042 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:28:38.043 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-13T16:28:38.043 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:28:38.043 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T16:28:38.044 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:28:38.044 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-13T16:28:38.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:28:31.929537+0000 '>' 2024-09-13T16:28:05.763771+0000 2024-09-13T16:28:38.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T16:28:38.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:98: TEST_scrub_test: ceph pg dump pgs 2024-09-13T16:28:38.359 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:98: TEST_scrub_test: grep '^1.0' 2024-09-13T16:28:38.360 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:98: TEST_scrub_test: grep -q -- +inconsistent 2024-09-13T16:28:38.649 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:28:38.666 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:99: TEST_scrub_test: jq .info.stats.stat_sum.num_scrub_errors 2024-09-13T16:28:38.666 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:99: TEST_scrub_test: ceph pg 1.0 query 2024-09-13T16:28:38.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:99: TEST_scrub_test: test 2 = 2 2024-09-13T16:28:38.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:101: TEST_scrub_test: ceph osd out 1 2024-09-13T16:28:39.239 INFO:tasks.workunit.client.0.smithi112.stderr:marked out osd.1. 2024-09-13T16:28:39.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:102: TEST_scrub_test: wait_for_clean 2024-09-13T16:28:39.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:28:39.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:28:39.260 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:28:39.260 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:28:39.261 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:28:39.261 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:28:39.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:28:39.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:28:39.261 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:28:39.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:28:39.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:28:39.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:28:39.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:28:39.375 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:28:39.375 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:28:39.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:28:39.896 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:28:39.896 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:28:39.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:28:39.896 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:28:39.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:28:40.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836517 2024-09-13T16:28:40.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836517 2024-09-13T16:28:40.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517' 2024-09-13T16:28:40.079 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:28:40.079 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:28:40.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672988 2024-09-13T16:28:40.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672988 2024-09-13T16:28:40.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-42949672988' 2024-09-13T16:28:40.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:28:40.219 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:28:40.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=98784247814 2024-09-13T16:28:40.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 98784247814 2024-09-13T16:28:40.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836517 1-42949672988 2-98784247814' 2024-09-13T16:28:40.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:28:40.437 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836517 2024-09-13T16:28:40.438 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:28:40.439 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:28:40.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836517 2024-09-13T16:28:40.440 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:28:40.441 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836517 2024-09-13T16:28:40.441 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836517 2024-09-13T16:28:40.442 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-09-13T16:28:40.442 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:28:40.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836516 -lt 21474836517 2024-09-13T16:28:40.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:28:41.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:28:41.896 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:28:42.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836517 -lt 21474836517 2024-09-13T16:28:42.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:28:42.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672988 2024-09-13T16:28:42.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:28:42.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:28:42.220 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672988 2024-09-13T16:28:42.220 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:28:42.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672988 2024-09-13T16:28:42.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-09-13T16:28:42.221 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672988 2024-09-13T16:28:42.222 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:28:42.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672988 -lt 42949672988 2024-09-13T16:28:42.531 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:28:42.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-98784247814 2024-09-13T16:28:42.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:28:42.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:28:42.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-98784247814 2024-09-13T16:28:42.535 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:28:42.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=98784247814 2024-09-13T16:28:42.537 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 98784247814' 2024-09-13T16:28:42.537 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 98784247814 2024-09-13T16:28:42.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:28:42.880 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 98784247814 -lt 98784247814 2024-09-13T16:28:42.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:28:42.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:28:42.881 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:28:43.270 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:28:43.271 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:28:43.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:28:43.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:28:43.271 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:28:43.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:28:43.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:28:43.272 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:28:43.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T16:28:43.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:28:43.675 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:28:43.675 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:28:44.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T16:28:44.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T16:28:44.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T16:28:44.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T16:28:44.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T16:28:44.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T16:28:44.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T16:28:44.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:28:44.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:28:44.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:28:44.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:28:44.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:28:44.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:28:44.153 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:28:44.465 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T16:28:44.465 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:28:44.466 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:28:44.466 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:28:44.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T16:28:44.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T16:28:44.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T16:28:44.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T16:28:44.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T16:28:44.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T16:28:44.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T16:28:44.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T16:28:44.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T16:28:45.200 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T16:28:45.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T16:28:45.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T16:28:45.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T16:28:45.201 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T16:28:45.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T16:28:45.402 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:28:45.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:28:45.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:28:45.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:28:45.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:28:45.403 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:28:45.404 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:28:45.729 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:28:45.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:28:45.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:28:45.730 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:28:46.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:28:46.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:28:46.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:28:46.103 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:104: TEST_scrub_test: pg_deep_scrub 1.0 2024-09-13T16:28:46.104 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=1.0 2024-09-13T16:28:46.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-13T16:28:46.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:28:46.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T16:28:46.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:28:46.105 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-13T16:28:46.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-13T16:28:31.929537+0000 2024-09-13T16:28:46.394 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 1.0 2024-09-13T16:28:46.690 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.0 on osd.0 to deep-scrub 2024-09-13T16:28:46.706 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 1.0 2024-09-13T16:28:31.929537+0000 last_deep_scrub_stamp 2024-09-13T16:28:46.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T16:28:46.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T16:28:31.929537+0000 2024-09-13T16:28:46.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-13T16:28:46.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T16:28:46.707 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:28:46.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-13T16:28:46.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:28:46.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T16:28:46.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:28:46.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-13T16:28:47.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:28:31.929537+0000 '>' 2024-09-13T16:28:31.929537+0000 2024-09-13T16:28:47.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:28:48.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:28:48.008 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:28:48.008 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-13T16:28:48.008 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:28:48.008 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T16:28:48.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:28:48.009 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-13T16:28:48.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:28:31.929537+0000 '>' 2024-09-13T16:28:31.929537+0000 2024-09-13T16:28:48.380 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:28:49.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:28:49.382 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:28:49.383 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2024-09-13T16:28:49.383 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:28:49.383 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-13T16:28:49.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:28:49.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2024-09-13T16:28:49.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:28:47.525957+0000 '>' 2024-09-13T16:28:31.929537+0000 2024-09-13T16:28:49.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T16:28:49.717 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:106: TEST_scrub_test: ceph pg 1.0 query 2024-09-13T16:28:49.717 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:106: TEST_scrub_test: jq .info.stats.stat_sum.num_scrub_errors 2024-09-13T16:28:49.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:106: TEST_scrub_test: test 2 = 2 2024-09-13T16:28:49.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:107: TEST_scrub_test: ceph pg 1.0 query 2024-09-13T16:28:49.860 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:107: TEST_scrub_test: jq '.peer_info[0].stats.stat_sum.num_scrub_errors' 2024-09-13T16:28:50.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:107: TEST_scrub_test: test 2 = 2 2024-09-13T16:28:50.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:108: TEST_scrub_test: ceph pg dump pgs 2024-09-13T16:28:50.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:108: TEST_scrub_test: grep '^1.0' 2024-09-13T16:28:50.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:108: TEST_scrub_test: grep -q -- +inconsistent 2024-09-13T16:28:50.289 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:28:50.305 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:110: TEST_scrub_test: ceph osd in 1 2024-09-13T16:28:50.828 INFO:tasks.workunit.client.0.smithi112.stderr:marked in osd.1. 2024-09-13T16:28:50.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:111: TEST_scrub_test: wait_for_clean 2024-09-13T16:28:50.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:28:50.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:28:50.847 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:28:50.847 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:28:50.848 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:28:50.848 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:28:50.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:28:50.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:28:50.849 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:28:50.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:28:50.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:28:50.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:28:50.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:28:50.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:28:50.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:28:51.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:28:51.510 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:28:51.510 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:28:51.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:28:51.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:28:51.511 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:28:51.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836520 2024-09-13T16:28:51.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836520 2024-09-13T16:28:51.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520' 2024-09-13T16:28:51.709 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:28:51.709 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:28:51.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672992 2024-09-13T16:28:51.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672992 2024-09-13T16:28:51.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520 1-42949672992' 2024-09-13T16:28:51.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:28:51.873 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:28:52.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=98784247817 2024-09-13T16:28:52.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 98784247817 2024-09-13T16:28:52.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836520 1-42949672992 2-98784247817' 2024-09-13T16:28:52.187 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:28:52.187 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836520 2024-09-13T16:28:52.188 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:28:52.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:28:52.190 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836520 2024-09-13T16:28:52.190 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:28:52.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836520 2024-09-13T16:28:52.191 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-09-13T16:28:52.191 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836520 2024-09-13T16:28:52.192 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:28:52.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836519 -lt 21474836520 2024-09-13T16:28:52.517 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:28:53.518 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:28:53.519 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:28:53.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836520 -lt 21474836520 2024-09-13T16:28:53.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:28:53.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672992 2024-09-13T16:28:53.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:28:53.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:28:53.855 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672992 2024-09-13T16:28:53.856 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:28:53.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672992 2024-09-13T16:28:53.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-09-13T16:28:53.857 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672992 2024-09-13T16:28:53.860 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:28:54.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672992 -lt 42949672992 2024-09-13T16:28:54.165 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:28:54.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-98784247817 2024-09-13T16:28:54.166 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:28:54.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:28:54.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:28:54.170 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-98784247817 2024-09-13T16:28:54.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=98784247817 2024-09-13T16:28:54.171 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 98784247817 2024-09-13T16:28:54.171 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 98784247817' 2024-09-13T16:28:54.171 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:28:54.525 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 98784247817 -lt 98784247817 2024-09-13T16:28:54.525 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:28:54.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:28:54.526 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:28:54.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:28:54.899 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:28:54.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:28:54.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:28:54.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:28:54.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:28:54.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:28:54.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:28:55.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T16:28:55.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:28:55.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:28:55.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:28:55.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T16:28:55.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T16:28:55.611 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T16:28:55.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T16:28:55.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T16:28:55.612 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T16:28:55.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T16:28:55.713 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:28:55.713 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:28:55.714 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:28:55.714 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:28:55.714 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:28:55.714 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:28:55.715 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:28:56.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T16:28:56.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:28:56.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:28:56.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:28:56.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T16:28:56.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T16:28:56.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T16:28:56.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T16:28:56.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T16:28:56.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T16:28:56.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T16:28:56.384 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T16:28:56.385 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T16:28:56.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T16:28:56.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T16:28:56.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T16:28:56.813 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T16:28:56.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T16:28:57.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T16:28:57.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:28:57.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:28:57.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:28:57.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:28:57.015 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:28:57.016 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:28:57.016 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:28:57.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:28:57.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:28:57.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:28:57.321 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:28:57.722 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:28:57.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:28:57.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:28:57.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:113: TEST_scrub_test: repair 1.0 2024-09-13T16:28:57.723 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-13T16:28:57.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-13T16:28:57.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:28:57.723 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:28:57.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:28:57.724 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:28:58.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-13T16:28:47.525957+0000 2024-09-13T16:28:58.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-13T16:28:58.315 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.0 on osd.1 to repair 2024-09-13T16:28:58.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-13T16:28:47.525957+0000 2024-09-13T16:28:58.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-13T16:28:58.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-13T16:28:47.525957+0000 2024-09-13T16:28:58.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-13T16:28:58.331 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-13T16:28:58.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:28:58.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:28:58.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:28:58.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:28:58.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:28:58.332 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:28:58.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:28:47.525957+0000 '>' 2024-09-13T16:28:47.525957+0000 2024-09-13T16:28:58.651 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:28:59.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:28:59.652 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:28:59.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:28:59.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:28:59.653 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:28:59.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:28:59.654 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:28:59.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:28:47.525957+0000 '>' 2024-09-13T16:28:47.525957+0000 2024-09-13T16:28:59.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:29:00.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:29:00.966 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:29:00.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:29:00.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:29:00.967 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:29:00.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:29:00.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:29:01.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:28:47.525957+0000 '>' 2024-09-13T16:28:47.525957+0000 2024-09-13T16:29:01.295 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:29:02.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:29:02.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:29:02.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:29:02.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:29:02.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:29:02.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:29:02.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:29:02.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:28:47.525957+0000 '>' 2024-09-13T16:28:47.525957+0000 2024-09-13T16:29:02.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-13T16:29:03.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-13T16:29:03.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 500 )) 2024-09-13T16:29:03.632 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-13T16:29:03.632 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-13T16:29:03.632 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-13T16:29:03.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-13T16:29:03.633 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-13T16:29:03.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-13T16:28:59.994179+0000 '>' 2024-09-13T16:28:47.525957+0000 2024-09-13T16:29:03.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-13T16:29:03.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-13T16:29:05.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:114: TEST_scrub_test: wait_for_clean 2024-09-13T16:29:05.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:29:05.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:29:05.957 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:29:05.957 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:29:05.958 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:29:05.958 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:29:05.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:29:05.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:29:05.958 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:29:06.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:29:06.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:29:06.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:29:06.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:29:06.085 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:29:06.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:29:06.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:29:06.413 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:29:06.413 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:29:06.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:29:06.413 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:29:06.413 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:29:06.559 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836524 2024-09-13T16:29:06.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836524 2024-09-13T16:29:06.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524' 2024-09-13T16:29:06.560 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:29:06.560 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:29:06.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672996 2024-09-13T16:29:06.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672996 2024-09-13T16:29:06.695 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949672996' 2024-09-13T16:29:06.696 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:29:06.696 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:29:06.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=98784247821 2024-09-13T16:29:06.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 98784247821 2024-09-13T16:29:06.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-98784247821' 2024-09-13T16:29:06.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:29:06.834 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836524 2024-09-13T16:29:06.834 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:29:06.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:29:06.835 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836524 2024-09-13T16:29:06.835 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:29:06.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836524 2024-09-13T16:29:06.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-13T16:29:06.837 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836524 2024-09-13T16:29:06.837 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:29:07.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836523 -lt 21474836524 2024-09-13T16:29:07.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:29:08.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:29:08.132 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:29:08.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-13T16:29:08.445 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:29:08.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672996 2024-09-13T16:29:08.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:29:08.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:29:08.449 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672996 2024-09-13T16:29:08.450 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:29:08.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672996 2024-09-13T16:29:08.451 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672996 2024-09-13T16:29:08.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-09-13T16:29:08.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:29:08.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672996 -lt 42949672996 2024-09-13T16:29:08.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:29:08.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-98784247821 2024-09-13T16:29:08.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:29:08.805 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:29:08.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-98784247821 2024-09-13T16:29:08.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:29:08.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=98784247821 2024-09-13T16:29:08.807 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 98784247821' 2024-09-13T16:29:08.807 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 98784247821 2024-09-13T16:29:08.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:29:09.185 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 98784247821 -lt 98784247821 2024-09-13T16:29:09.185 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:29:09.186 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:29:09.186 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:29:09.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:29:09.673 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:29:09.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:29:09.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:29:09.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:29:09.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:29:09.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:29:09.674 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:29:09.986 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:29:09.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:29:09.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:29:09.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:29:10.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:29:10.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:29:10.361 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:29:10.361 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:117: TEST_scrub_test: ceph pg 1.0 query 2024-09-13T16:29:10.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:117: TEST_scrub_test: jq '.peer_info[0].stats.stat_sum.num_scrub_errors' 2024-09-13T16:29:10.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:117: TEST_scrub_test: test 2 = 2 2024-09-13T16:29:10.555 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:118: TEST_scrub_test: ceph pg dump pgs 2024-09-13T16:29:10.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:118: TEST_scrub_test: grep '^1.0' 2024-09-13T16:29:10.556 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:118: TEST_scrub_test: grep -vq -- +inconsistent 2024-09-13T16:29:10.885 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:29:10.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:120: TEST_scrub_test: ceph osd out 1 2024-09-13T16:29:11.336 INFO:tasks.workunit.client.0.smithi112.stderr:marked out osd.1. 2024-09-13T16:29:11.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: TEST_scrub_test: wait_for_clean 2024-09-13T16:29:11.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:29:11.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:29:11.353 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:29:11.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:29:11.354 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:29:11.354 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:29:11.354 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:29:11.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:29:11.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:29:11.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:29:11.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:29:11.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:29:11.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:29:11.461 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=500 2024-09-13T16:29:11.461 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:29:12.065 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:29:12.065 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:29:12.065 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:29:12.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:29:12.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:29:12.066 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:29:12.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836526 2024-09-13T16:29:12.261 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836526 2024-09-13T16:29:12.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526' 2024-09-13T16:29:12.262 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:29:12.262 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:29:12.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672998 2024-09-13T16:29:12.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672998 2024-09-13T16:29:12.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-42949672998' 2024-09-13T16:29:12.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:29:12.571 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:29:12.786 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=98784247823 2024-09-13T16:29:12.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 98784247823 2024-09-13T16:29:12.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-42949672998 2-98784247823' 2024-09-13T16:29:12.787 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:29:12.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836526 2024-09-13T16:29:12.787 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:29:12.788 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:29:12.789 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836526 2024-09-13T16:29:12.789 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:29:12.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836526 2024-09-13T16:29:12.790 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-09-13T16:29:12.791 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836526 2024-09-13T16:29:12.791 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:29:13.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836525 -lt 21474836526 2024-09-13T16:29:13.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:29:14.223 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 500 -eq 0 ']' 2024-09-13T16:29:14.224 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:29:14.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836526 -lt 21474836526 2024-09-13T16:29:14.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:29:14.536 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672998 2024-09-13T16:29:14.537 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:29:14.538 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:29:14.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672998 2024-09-13T16:29:14.539 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:29:14.541 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672998 2024-09-13T16:29:14.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672998 2024-09-13T16:29:14.541 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-09-13T16:29:14.542 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:29:14.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672998 -lt 42949672998 2024-09-13T16:29:14.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:29:14.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-98784247823 2024-09-13T16:29:14.855 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:29:14.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:29:14.857 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-98784247823 2024-09-13T16:29:14.857 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:29:14.858 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=98784247823 2024-09-13T16:29:14.859 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 98784247823 2024-09-13T16:29:14.859 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 98784247823' 2024-09-13T16:29:14.859 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:29:15.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 98784247823 -lt 98784247823 2024-09-13T16:29:15.187 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:29:15.187 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:29:15.187 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:29:15.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:29:15.657 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:29:15.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:29:15.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:29:15.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:29:15.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:29:15.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:29:15.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:29:15.965 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:29:15.965 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:29:15.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:29:15.966 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:29:16.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:29:16.349 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:29:16.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:29:16.350 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:123: TEST_scrub_test: ceph pg 1.0 query 2024-09-13T16:29:16.350 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:123: TEST_scrub_test: jq .info.stats.stat_sum.num_scrub_errors 2024-09-13T16:29:16.532 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:123: TEST_scrub_test: test 0 = 0 2024-09-13T16:29:16.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: TEST_scrub_test: ceph pg 1.0 query 2024-09-13T16:29:16.532 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: TEST_scrub_test: jq '.peer_info[0].stats.stat_sum.num_scrub_errors' 2024-09-13T16:29:16.665 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: TEST_scrub_test: test 0 = 0 2024-09-13T16:29:16.666 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: TEST_scrub_test: jq '.peer_info[1].stats.stat_sum.num_scrub_errors' 2024-09-13T16:29:16.666 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: TEST_scrub_test: ceph pg 1.0 query 2024-09-13T16:29:16.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:125: TEST_scrub_test: test 0 = 0 2024-09-13T16:29:16.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:126: TEST_scrub_test: ceph pg dump pgs 2024-09-13T16:29:16.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:126: TEST_scrub_test: grep '^1.0' 2024-09-13T16:29:16.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:126: TEST_scrub_test: grep -vq -- +inconsistent 2024-09-13T16:29:17.089 INFO:tasks.workunit.client.0.smithi112.stderr:dumped pgs 2024-09-13T16:29:17.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:127: TEST_scrub_test: perf_counters td/osd-scrub-test 3 2024-09-13T16:29:17.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-09-13T16:29:17.106 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2024-09-13T16:29:17.107 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2024-09-13T16:29:17.108 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2024-09-13T16:29:17.109 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:29:17.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-09-13T16:29:17.110 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:29:17.236 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:29:17.236 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:29:17.236 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:29:17.237 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:29:17.237 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:29:17.237 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:29:17.237 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.237 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:29:17.237 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:29:17.237 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:29:17.237 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:29:17.237 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:29:17.237 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.237 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.237 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.238 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.238 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:29:17.238 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:29:17.238 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.238 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.238 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.238 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.238 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:29:17.238 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:29:17.238 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:29:17.238 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:29:17.238 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:29:17.238 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:29:17.239 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:29:17.239 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.239 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.239 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.239 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.239 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:29:17.239 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:29:17.239 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:29:17.239 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:29:17.239 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.239 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.239 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.239 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.239 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:29:17.240 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.240 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.240 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:29:17.240 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:29:17.240 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:29:17.240 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:29:17.240 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:29:17.240 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:29:17.240 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.240 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:29:17.240 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 1, 2024-09-13T16:29:17.240 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 1, 2024-09-13T16:29:17.240 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 1, 2024-09-13T16:29:17.241 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:29:17.241 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 1, 2024-09-13T16:29:17.241 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0.119996959, 2024-09-13T16:29:17.241 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0.119996959 2024-09-13T16:29:17.241 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.241 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:29:17.241 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:29:17.241 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.241 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.241 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.241 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.241 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:29:17.241 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 1, 2024-09-13T16:29:17.241 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:29:17.242 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:29:17.242 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:29:17.242 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:29:17.242 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:29:17.242 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.242 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.242 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.242 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.242 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:29:17.242 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:29:17.242 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 1, 2024-09-13T16:29:17.242 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:29:17.242 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.243 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.243 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.243 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.243 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 2 2024-09-13T16:29:17.243 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.243 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.243 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:29:17.243 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:29:17.243 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:29:17.243 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:29:17.243 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:29:17.243 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:29:17.243 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.243 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:29:17.244 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:29:17.244 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:29:17.244 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:29:17.244 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:29:17.244 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.244 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.244 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.244 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.244 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:29:17.244 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:29:17.244 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.244 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.244 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.245 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.245 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:29:17.245 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:29:17.245 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:29:17.245 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:29:17.245 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:29:17.245 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:29:17.245 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:29:17.245 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.245 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.245 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.245 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.245 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:29:17.246 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:29:17.246 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:29:17.246 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:29:17.246 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.246 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.246 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.246 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.246 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:29:17.246 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.246 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.246 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:29:17.246 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:29:17.246 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:29:17.247 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:29:17.247 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:29:17.247 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:29:17.247 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.247 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:29:17.247 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:29:17.247 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:29:17.247 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:29:17.247 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:29:17.247 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.247 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.247 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.247 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.247 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:29:17.247 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:29:17.248 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.248 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.248 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.248 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.248 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:29:17.248 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:29:17.248 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:29:17.248 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:29:17.248 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:29:17.248 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:29:17.248 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:29:17.249 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.249 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.249 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.249 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.249 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:29:17.249 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:29:17.249 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:29:17.249 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:29:17.249 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.249 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.249 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.249 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.250 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:29:17.250 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.250 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.250 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:29:17.250 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:29:17.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:29:17.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-09-13T16:29:17.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:29:17.373 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:29:17.373 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:29:17.373 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:29:17.373 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:29:17.373 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:29:17.373 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:29:17.373 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.374 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:29:17.374 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:29:17.374 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:29:17.374 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:29:17.374 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:29:17.374 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.374 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.374 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.374 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.374 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:29:17.374 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:29:17.374 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.374 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.375 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.375 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.375 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:29:17.375 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:29:17.375 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:29:17.375 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:29:17.375 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:29:17.375 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:29:17.375 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:29:17.375 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.375 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.375 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.375 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.375 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:29:17.376 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:29:17.376 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:29:17.376 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:29:17.376 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.376 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.376 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.376 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.376 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:29:17.377 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.377 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.377 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:29:17.377 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:29:17.377 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:29:17.377 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:29:17.377 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:29:17.377 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:29:17.377 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.377 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:29:17.377 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 3, 2024-09-13T16:29:17.378 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 3, 2024-09-13T16:29:17.378 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 3, 2024-09-13T16:29:17.378 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:29:17.378 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 3, 2024-09-13T16:29:17.378 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0.235994023, 2024-09-13T16:29:17.378 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0.078664674 2024-09-13T16:29:17.379 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.379 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:29:17.379 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:29:17.379 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.379 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.379 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.379 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.379 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:29:17.379 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 3, 2024-09-13T16:29:17.380 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:29:17.380 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:29:17.380 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:29:17.380 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:29:17.380 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:29:17.380 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.380 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.380 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.381 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.381 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:29:17.381 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:29:17.381 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 3, 2024-09-13T16:29:17.381 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:29:17.381 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.381 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.381 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.381 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.381 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 2 2024-09-13T16:29:17.381 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.381 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.381 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:29:17.382 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:29:17.382 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:29:17.382 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:29:17.382 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:29:17.382 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:29:17.382 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.382 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:29:17.382 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:29:17.382 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:29:17.382 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:29:17.382 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:29:17.382 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.382 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.383 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.383 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.383 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:29:17.383 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:29:17.383 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.383 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.383 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.383 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.383 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:29:17.383 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:29:17.383 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:29:17.383 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:29:17.383 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:29:17.383 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:29:17.384 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:29:17.384 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.384 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.384 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.384 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.384 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:29:17.384 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:29:17.384 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:29:17.384 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:29:17.384 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.384 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.384 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.385 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.385 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:29:17.385 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.385 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.385 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:29:17.385 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:29:17.385 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:29:17.385 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:29:17.385 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:29:17.385 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:29:17.385 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.385 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:29:17.385 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:29:17.386 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:29:17.386 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:29:17.386 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:29:17.386 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.386 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.386 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.386 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.386 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:29:17.386 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:29:17.386 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.386 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.386 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.386 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.386 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:29:17.387 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:29:17.387 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:29:17.387 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:29:17.387 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:29:17.387 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:29:17.387 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:29:17.387 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.387 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.387 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.388 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.388 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:29:17.388 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:29:17.388 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:29:17.388 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:29:17.388 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.388 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.388 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.388 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.388 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:29:17.388 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.388 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.388 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:29:17.389 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:29:17.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-13T16:29:17.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-09-13T16:29:17.389 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2024-09-13T16:29:17.505 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:29:17.505 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_ec": [ 2024-09-13T16:29:17.506 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:29:17.506 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:29:17.506 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:29:17.506 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:29:17.506 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.506 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:29:17.506 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:29:17.506 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:29:17.506 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:29:17.506 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:29:17.506 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.506 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.506 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.506 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.507 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:29:17.507 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:29:17.507 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.507 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.507 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.507 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.507 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:29:17.507 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:29:17.507 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:29:17.507 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:29:17.507 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:29:17.507 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:29:17.507 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:29:17.507 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.508 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.508 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.508 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.508 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:29:17.508 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:29:17.508 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:29:17.508 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:29:17.508 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.508 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.508 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.509 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.509 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:29:17.509 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.509 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.509 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:29:17.509 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_dp_repl": [ 2024-09-13T16:29:17.509 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:29:17.509 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:29:17.509 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "deep", 2024-09-13T16:29:17.509 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:29:17.509 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.509 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:29:17.509 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:29:17.510 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:29:17.510 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:29:17.510 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:29:17.510 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.510 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.510 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.510 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.510 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:29:17.510 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:29:17.510 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.510 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.510 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.510 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.510 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:29:17.511 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:29:17.511 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:29:17.511 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:29:17.511 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:29:17.511 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:29:17.511 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:29:17.511 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.511 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.511 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.511 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.511 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:29:17.511 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:29:17.511 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:29:17.512 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:29:17.512 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.512 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.512 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.512 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.512 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:29:17.512 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.512 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.512 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:29:17.512 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_ec": [ 2024-09-13T16:29:17.512 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:29:17.512 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:29:17.512 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:29:17.512 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "ec" 2024-09-13T16:29:17.513 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.513 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:29:17.513 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:29:17.513 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:29:17.513 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:29:17.513 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:29:17.513 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.513 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.513 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.513 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.513 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:29:17.513 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:29:17.513 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.514 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.514 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.514 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.514 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:29:17.514 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:29:17.514 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:29:17.514 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:29:17.514 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:29:17.514 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:29:17.514 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:29:17.514 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.514 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.514 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.514 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.515 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:29:17.515 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:29:17.515 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:29:17.515 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:29:17.515 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.515 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.515 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.515 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.515 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:29:17.515 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.515 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.515 INFO:tasks.workunit.client.0.smithi112.stdout: ], 2024-09-13T16:29:17.515 INFO:tasks.workunit.client.0.smithi112.stdout: "osd_scrub_sh_repl": [ 2024-09-13T16:29:17.515 INFO:tasks.workunit.client.0.smithi112.stdout: { 2024-09-13T16:29:17.516 INFO:tasks.workunit.client.0.smithi112.stdout: "labels": { 2024-09-13T16:29:17.516 INFO:tasks.workunit.client.0.smithi112.stdout: "level": "shallow", 2024-09-13T16:29:17.516 INFO:tasks.workunit.client.0.smithi112.stdout: "pooltype": "replicated" 2024-09-13T16:29:17.516 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.516 INFO:tasks.workunit.client.0.smithi112.stdout: "counters": { 2024-09-13T16:29:17.516 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_started": 0, 2024-09-13T16:29:17.516 INFO:tasks.workunit.client.0.smithi112.stdout: "num_scrubs_past_reservation": 0, 2024-09-13T16:29:17.516 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs": 0, 2024-09-13T16:29:17.516 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_scrubs_elapsed": { 2024-09-13T16:29:17.516 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.516 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.516 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.516 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.516 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs": 0, 2024-09-13T16:29:17.517 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_scrubs_elapsed": { 2024-09-13T16:29:17.517 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.517 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.517 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.517 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.517 INFO:tasks.workunit.client.0.smithi112.stdout: "preemptions": 0, 2024-09-13T16:29:17.517 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_selected": 0, 2024-09-13T16:29:17.517 INFO:tasks.workunit.client.0.smithi112.stdout: "chunk_busy": 0, 2024-09-13T16:29:17.517 INFO:tasks.workunit.client.0.smithi112.stdout: "locked_object": 0, 2024-09-13T16:29:17.517 INFO:tasks.workunit.client.0.smithi112.stdout: "write_blocked_by_scrub": 0, 2024-09-13T16:29:17.517 INFO:tasks.workunit.client.0.smithi112.stdout: "scrub_reservations_completed": 0, 2024-09-13T16:29:17.517 INFO:tasks.workunit.client.0.smithi112.stdout: "successful_reservations_elapsed": { 2024-09-13T16:29:17.518 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.518 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.518 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.518 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.518 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_aborted": 0, 2024-09-13T16:29:17.518 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_failure": 0, 2024-09-13T16:29:17.518 INFO:tasks.workunit.client.0.smithi112.stdout: "reservation_process_skipped": 0, 2024-09-13T16:29:17.518 INFO:tasks.workunit.client.0.smithi112.stdout: "failed_reservations_elapsed": { 2024-09-13T16:29:17.518 INFO:tasks.workunit.client.0.smithi112.stdout: "avgcount": 0, 2024-09-13T16:29:17.519 INFO:tasks.workunit.client.0.smithi112.stdout: "sum": 0, 2024-09-13T16:29:17.519 INFO:tasks.workunit.client.0.smithi112.stdout: "avgtime": 0 2024-09-13T16:29:17.519 INFO:tasks.workunit.client.0.smithi112.stdout: }, 2024-09-13T16:29:17.519 INFO:tasks.workunit.client.0.smithi112.stdout: "replicas_in_reservation": 0 2024-09-13T16:29:17.519 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.519 INFO:tasks.workunit.client.0.smithi112.stdout: } 2024-09-13T16:29:17.519 INFO:tasks.workunit.client.0.smithi112.stdout: ] 2024-09-13T16:29:17.519 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:29:17.519 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Teardown Test TEST_scrub_test ------------------ 2024-09-13T16:29:17.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_scrub_test ------------------' 2024-09-13T16:29:17.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-09-13T16:29:17.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:29:17.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:29:17.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:29:17.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:29:17.520 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:29:17.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:29:17.520 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:29:17.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:29:17.828 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:29:17.828 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:29:17.829 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:29:17.830 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:29:17.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:29:17.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:29:17.833 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:29:17.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:29:17.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:29:17.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:29:17.835 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:29:17.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:29:17.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:29:17.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:29:17.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:29:17.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:29:17.861 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:29:17.862 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:29:17.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:29:17.863 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:29:17.863 INFO:tasks.workunit.client.0.smithi112.stdout:-------------- Complete Test TEST_scrub_test ------------------ 2024-09-13T16:29:17.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_scrub_test ------------------' 2024-09-13T16:29:17.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-13T16:29:17.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-scrub-test 0 2024-09-13T16:29:17.864 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2024-09-13T16:29:17.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-13T16:29:17.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2024-09-13T16:29:17.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:29:17.865 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:29:17.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:29:17.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:29:17.865 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:29:17.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:29:17.868 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:29:17.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:29:17.869 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:29:17.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:29:17.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:29:17.870 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:29:17.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:29:17.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:29:17.872 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:29:17.872 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:29:17.873 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:29:17.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-13T16:29:17.874 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2024-09-13T16:29:17.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:29:17.875 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:29:17.876 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.528094 2024-09-13T16:29:17.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.528094 2024-09-13T16:29:17.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:29:17.876 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:29:17.877 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-13T16:29:17.878 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T16:29:17.878 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T16:29:17.934 INFO:tasks.workunit:Running workunit scrub/osd-unexpected-clone.sh... 2024-09-13T16:29:17.934 DEBUG:teuthology.orchestra.run.smithi112:workunit test scrub/osd-unexpected-clone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d58f73069a8157a9ace12a0247ee3e8497707ea9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh 2024-09-13T16:29:17.990 INFO:tasks.workunit.client.0.smithi112.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-13T16:29:17.996 INFO:tasks.workunit.client.0.smithi112.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-13T16:29:17.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-13T16:29:17.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-13T16:29:17.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-13T16:29:17.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-13T16:29:17.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-13T16:29:17.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-13T16:29:17.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-13T16:29:17.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-13T16:29:17.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-unexpected-clone 2024-09-13T16:29:17.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:22: run: local dir=td/osd-unexpected-clone 2024-09-13T16:29:17.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:23: run: shift 2024-09-13T16:29:17.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:25: run: export CEPH_MON=127.0.0.1:7144 2024-09-13T16:29:17.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:25: run: CEPH_MON=127.0.0.1:7144 2024-09-13T16:29:17.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:26: run: export CEPH_ARGS 2024-09-13T16:29:17.997 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:27: run: uuidgen 2024-09-13T16:29:17.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:27: run: CEPH_ARGS+='--fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none ' 2024-09-13T16:29:17.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-09-13T16:29:17.998 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:30: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-13T16:29:17.998 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:31: run: set 2024-09-13T16:29:17.998 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-13T16:29:17.999 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:31: run: local funcs=TEST_recover_unexpected 2024-09-13T16:29:18.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:32: run: for func in $funcs 2024-09-13T16:29:18.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:33: run: setup td/osd-unexpected-clone 2024-09-13T16:29:18.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-unexpected-clone 2024-09-13T16:29:18.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-unexpected-clone 2024-09-13T16:29:18.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-unexpected-clone 2024-09-13T16:29:18.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:29:18.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-unexpected-clone KILL 2024-09-13T16:29:18.000 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:29:18.000 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:29:18.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:29:18.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:29:18.000 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:29:18.002 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:29:18.003 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:29:18.003 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:29:18.004 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:29:18.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:29:18.005 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:29:18.005 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:29:18.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:29:18.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:29:18.006 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:29:18.007 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:29:18.007 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:29:18.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:29:18.009 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-unexpected-clone 2024-09-13T16:29:18.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:29:18.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:29:18.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:29:18.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.643418 2024-09-13T16:29:18.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:29:18.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:29:18.012 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-unexpected-clone 2024-09-13T16:29:18.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-13T16:29:18.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:29:18.013 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:29:18.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.643418 2024-09-13T16:29:18.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-13T16:29:18.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-13T16:29:18.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-13T16:29:18.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-13T16:29:18.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-unexpected-clone 1' TERM HUP INT 2024-09-13T16:29:18.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:34: run: TEST_recover_unexpected td/osd-unexpected-clone 2024-09-13T16:29:18.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:40: TEST_recover_unexpected: local dir=td/osd-unexpected-clone 2024-09-13T16:29:18.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:42: TEST_recover_unexpected: run_mon td/osd-unexpected-clone a 2024-09-13T16:29:18.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-unexpected-clone 2024-09-13T16:29:18.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-13T16:29:18.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-13T16:29:18.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-13T16:29:18.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-unexpected-clone/a 2024-09-13T16:29:18.017 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-unexpected-clone/a --run-dir=td/osd-unexpected-clone 2024-09-13T16:29:18.552 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-13T16:29:18.552 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:29:18.552 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:29:18.553 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:29:18.553 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:29:18.553 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:29:18.553 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:29:18.553 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-unexpected-clone/a '--log-file=td/osd-unexpected-clone/$name.log' '--admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' --mon-cluster-log-file=td/osd-unexpected-clone/log --run-dir=td/osd-unexpected-clone '--pid-file=td/osd-unexpected-clone/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-13T16:29:18.798 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-13T16:29:18.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-13T16:29:18.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T16:29:18.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T16:29:18.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-13T16:29:18.800 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-13T16:29:18.800 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T16:29:18.800 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T16:29:18.800 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T16:29:18.800 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:29:18.800 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:29:18.800 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:29:18.800 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.643418/ceph-mon.a.asok 2024-09-13T16:29:18.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T16:29:18.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.643418/ceph-mon.a.asok config get fsid 2024-09-13T16:29:18.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-13T16:29:18.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-13T16:29:18.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-13T16:29:18.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-13T16:29:18.918 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-13T16:29:18.918 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-13T16:29:18.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-13T16:29:18.918 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-13T16:29:18.918 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-13T16:29:18.918 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:29:18.918 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:29:18.918 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.643418/ceph-mon.a.asok 2024-09-13T16:29:18.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-13T16:29:18.919 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.643418/ceph-mon.a.asok config get mon_host 2024-09-13T16:29:19.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:43: TEST_recover_unexpected: run_mgr td/osd-unexpected-clone x 2024-09-13T16:29:19.028 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-unexpected-clone 2024-09-13T16:29:19.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-13T16:29:19.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-13T16:29:19.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-13T16:29:19.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-unexpected-clone/x 2024-09-13T16:29:19.029 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-13T16:29:20.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-13T16:29:20.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:29:20.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:29:20.756 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:29:20.756 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:29:20.756 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:29:20.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:29:20.756 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T16:29:20.757 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-unexpected-clone/x '--log-file=td/osd-unexpected-clone/$name.log' '--admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' --run-dir=td/osd-unexpected-clone '--pid-file=td/osd-unexpected-clone/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-13T16:29:20.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:44: TEST_recover_unexpected: run_osd td/osd-unexpected-clone 0 2024-09-13T16:29:20.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-unexpected-clone 2024-09-13T16:29:20.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:29:20.792 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-13T16:29:20.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:29:20.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-unexpected-clone/0 2024-09-13T16:29:20.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-13T16:29:20.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:29:20.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:29:20.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:29:20.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/0' 2024-09-13T16:29:20.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/0/journal' 2024-09-13T16:29:20.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:29:20.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:29:20.793 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-13T16:29:20.793 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:29:20.793 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:29:20.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:29:20.794 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:29:20.794 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:29:20.794 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:29:20.794 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:29:20.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:29:20.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:29:20.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:29:20.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:29:20.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-13T16:29:20.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-13T16:29:20.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:29:20.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:29:20.794 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:29:20.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:29:20.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:29:20.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T16:29:20.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-unexpected-clone/0 2024-09-13T16:29:20.795 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:29:20.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6cd3b30a-21bc-4507-ac00-c7c2b4ae5733 2024-09-13T16:29:20.795 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 6cd3b30a-21bc-4507-ac00-c7c2b4ae5733' 2024-09-13T16:29:20.796 INFO:tasks.workunit.client.0.smithi112.stdout:add osd0 6cd3b30a-21bc-4507-ac00-c7c2b4ae5733 2024-09-13T16:29:20.796 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:29:20.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBgaORm6jluMBAA1hjVfN7iDSvOZIZd7YKajw== 2024-09-13T16:29:20.811 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBgaORm6jluMBAA1hjVfN7iDSvOZIZd7YKajw=="}' 2024-09-13T16:29:20.812 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6cd3b30a-21bc-4507-ac00-c7c2b4ae5733 -i td/osd-unexpected-clone/0/new.json 2024-09-13T16:29:21.116 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:29:21.133 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-unexpected-clone/0/new.json 2024-09-13T16:29:21.134 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/0 --osd-journal=td/osd-unexpected-clone/0/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBgaORm6jluMBAA1hjVfN7iDSvOZIZd7YKajw== --osd-uuid 6cd3b30a-21bc-4507-ac00-c7c2b4ae5733 2024-09-13T16:29:21.165 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:29:21.162+0000 7f083bab37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:29:21.200 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:29:21.198+0000 7f083bab37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:29:21.202 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:29:21.202+0000 7f083bab37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:29:21.202 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:29:21.202+0000 7f083bab37c0 -1 bdev(0x55dfb9869c00 td/osd-unexpected-clone/0/block) open stat got: (1) Operation not permitted 2024-09-13T16:29:21.202 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:29:21.202+0000 7f083bab37c0 -1 bluestore(td/osd-unexpected-clone/0) _read_fsid unparsable uuid 2024-09-13T16:29:29.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-unexpected-clone/0/keyring 2024-09-13T16:29:29.327 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:29:29.328 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd0 key to auth repository 2024-09-13T16:29:29.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-13T16:29:29.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-unexpected-clone/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:29:29.725 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.0 2024-09-13T16:29:29.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-13T16:29:29.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/0 --osd-journal=td/osd-unexpected-clone/0/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T16:29:29.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:29:29.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:29:29.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:29:29.750 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:29:29.750+0000 7fcea1a627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:29:29.752 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:29:29.750+0000 7fcea1a627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:29:29.753 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:29:29.750+0000 7fcea1a627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:29:30.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-13T16:29:30.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:29:30.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-13T16:29:30.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:29:30.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:29:30.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:30.072 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:29:30.073 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:29:30.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:30.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:30.372 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:31.056 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:29:31.054+0000 7fcea1a627c0 -1 Falling back to public interface 2024-09-13T16:29:31.373 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:31.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:31.374 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:29:31.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:29:31.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:31.374 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:31.697 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:32.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:32.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:32.698 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:29:32.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:29:32.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:32.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:33.013 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:34.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:34.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:34.015 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:29:34.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:29:34.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:34.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:34.035 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:29:34.034+0000 7fcea1a627c0 -1 osd.0 0 log_to_monitors true 2024-09-13T16:29:34.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:35.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:35.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:35.406 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:29:35.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:29:35.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:35.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:35.731 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:36.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:36.733 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:36.733 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:29:36.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:29:36.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:36.734 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:37.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:38.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:38.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:38.249 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:29:38.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T16:29:38.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:38.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:38.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:39.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:39.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:39.581 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:29:39.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T16:29:39.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:39.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:39.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:40.906 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:40.906 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:29:40.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:40.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T16:29:40.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:40.907 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:41.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:42.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:42.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:42.301 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:29:42.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T16:29:42.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:42.302 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:42.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:43.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:43.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:43.630 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:29:43.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T16:29:43.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:43.631 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:44.024 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:45.025 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:45.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:45.026 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:29:45.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T16:29:45.026 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:45.027 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:45.354 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:46.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:46.356 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:46.356 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:29:46.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T16:29:46.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:46.357 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:46.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:47.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:47.742 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:29:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T16:29:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:47.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:48.247 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:49.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:49.249 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:49.250 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:29:49.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T16:29:49.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:49.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:49.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:50.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:50.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:50.574 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:29:50.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T16:29:50.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:50.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:50.902 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:51.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:51.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:51.904 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:29:51.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T16:29:51.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:51.905 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:52.231 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:53.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:53.233 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:53.233 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:29:53.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T16:29:53.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:53.234 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:53.970 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:54.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:54.972 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:29:54.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:54.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T16:29:54.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:54.972 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:55.398 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:56.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:56.399 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:56.399 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:29:56.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T16:29:56.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:56.400 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:57.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:58.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:58.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:58.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T16:29:58.126 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:29:58.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:58.127 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:29:58.806 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:29:59.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:29:59.808 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:29:59.809 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:29:59.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T16:29:59.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:29:59.809 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:30:00.129 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:01.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:01.131 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:01.131 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:30:01.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T16:30:01.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:01.132 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:30:01.949 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:02.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:02.951 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:02.951 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:30:02.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T16:30:02.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:02.952 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:30:03.589 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:04.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:04.591 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:04.591 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:30:04.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T16:30:04.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:04.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:30:05.436 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:06.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:06.437 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:06.438 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T16:30:06.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T16:30:06.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:06.438 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-13T16:30:06.797 INFO:tasks.workunit.client.0.smithi112.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2145697610,v1:127.0.0.1:6803/2145697610] [v2:127.0.0.1:6804/2145697610,v1:127.0.0.1:6805/2145697610] exists,up 6cd3b30a-21bc-4507-ac00-c7c2b4ae5733 2024-09-13T16:30:06.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:30:06.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:30:06.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:30:06.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:45: TEST_recover_unexpected: run_osd td/osd-unexpected-clone 1 2024-09-13T16:30:06.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-unexpected-clone 2024-09-13T16:30:06.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:30:06.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-13T16:30:06.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:30:06.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-unexpected-clone/1 2024-09-13T16:30:06.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-13T16:30:06.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:30:06.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:30:06.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:30:06.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/1' 2024-09-13T16:30:06.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/1/journal' 2024-09-13T16:30:06.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:30:06.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:30:06.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-13T16:30:06.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:30:06.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:30:06.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:30:06.802 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:30:06.802 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:30:06.802 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:30:06.802 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:30:06.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:30:06.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:30:06.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:30:06.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:30:06.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-13T16:30:06.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-13T16:30:06.802 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:30:06.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:30:06.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:30:06.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:30:06.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:30:06.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T16:30:06.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-unexpected-clone/1 2024-09-13T16:30:06.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:30:06.803 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2299a6cb-852f-4a5b-a91f-0f7d58220942 2024-09-13T16:30:06.804 INFO:tasks.workunit.client.0.smithi112.stdout:add osd1 2299a6cb-852f-4a5b-a91f-0f7d58220942 2024-09-13T16:30:06.804 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 2299a6cb-852f-4a5b-a91f-0f7d58220942' 2024-09-13T16:30:06.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:30:06.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCOaORmduklMRAAAK++7/ClGpF4s4C46gp/tw== 2024-09-13T16:30:06.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCOaORmduklMRAAAK++7/ClGpF4s4C46gp/tw=="}' 2024-09-13T16:30:06.823 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2299a6cb-852f-4a5b-a91f-0f7d58220942 -i td/osd-unexpected-clone/1/new.json 2024-09-13T16:30:07.287 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:30:07.308 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-unexpected-clone/1/new.json 2024-09-13T16:30:07.309 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCOaORmduklMRAAAK++7/ClGpF4s4C46gp/tw== --osd-uuid 2299a6cb-852f-4a5b-a91f-0f7d58220942 2024-09-13T16:30:07.342 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:30:07.341+0000 7fe065bc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:30:07.422 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:30:07.421+0000 7fe065bc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:30:07.424 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:30:07.421+0000 7fe065bc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:30:07.425 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:30:07.421+0000 7fe065bc47c0 -1 bdev(0x55cad9d72800 td/osd-unexpected-clone/1/block) open stat got: (1) Operation not permitted 2024-09-13T16:30:07.425 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:30:07.421+0000 7fe065bc47c0 -1 bluestore(td/osd-unexpected-clone/1) _read_fsid unparsable uuid 2024-09-13T16:30:16.124 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-unexpected-clone/1/keyring 2024-09-13T16:30:16.125 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:30:16.125 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd1 key to auth repository 2024-09-13T16:30:16.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-13T16:30:16.126 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-unexpected-clone/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:30:16.534 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-13T16:30:16.535 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.1 2024-09-13T16:30:16.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T16:30:16.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:30:16.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:30:16.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:30:16.564 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:30:16.561+0000 7fa453f147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:30:16.566 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:30:16.565+0000 7fa453f147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:30:16.568 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:30:16.565+0000 7fa453f147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:30:16.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-13T16:30:16.848 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:30:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T16:30:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:30:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:30:16.849 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:16.849 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:30:16.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:30:16.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:16.850 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:17.168 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:17.872 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:30:17.869+0000 7fa453f147c0 -1 Falling back to public interface 2024-09-13T16:30:18.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:18.169 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:18.169 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:30:18.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:30:18.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:18.170 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:18.488 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:19.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:19.489 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:19.490 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:30:19.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:30:19.490 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:19.491 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:19.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:20.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:20.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:20.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:30:20.827 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:30:20.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:20.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:21.123 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:30:21.121+0000 7fa453f147c0 -1 osd.1 0 log_to_monitors true 2024-09-13T16:30:21.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:22.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:22.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:22.158 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:30:22.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:30:22.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:22.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:22.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:23.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:23.605 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:23.606 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:30:23.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:30:23.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:23.606 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:24.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:25.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:25.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:25.058 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:30:25.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T16:30:25.058 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:25.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:25.449 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:26.451 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:26.451 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:30:26.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:26.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T16:30:26.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:26.452 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:26.797 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:27.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:27.799 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:27.799 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:30:27.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T16:30:27.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:27.800 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:28.248 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:29.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:29.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:29.250 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:30:29.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T16:30:29.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:29.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:29.582 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:30.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:30.583 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:30.583 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:30:30.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T16:30:30.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:30.584 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:30.910 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:31.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:31.911 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:31.911 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:30:31.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T16:30:31.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:31.912 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:32.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:33.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:33.240 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:33.240 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:30:33.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T16:30:33.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:33.241 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:33.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:34.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:34.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:34.571 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:30:34.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T16:30:34.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:34.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:34.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:35.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:35.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:35.891 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T16:30:35.891 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:30:35.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:35.892 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:36.330 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:37.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:37.332 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:37.332 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:30:37.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T16:30:37.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:37.333 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:37.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:38.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:38.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:38.743 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:30:38.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T16:30:38.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:38.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:39.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:40.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:40.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:40.076 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:30:40.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T16:30:40.076 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:40.077 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:40.748 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:41.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:41.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:41.750 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:30:41.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T16:30:41.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:41.751 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:42.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:43.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:43.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:43.510 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:30:43.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T16:30:43.510 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:43.511 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:44.277 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:45.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:45.278 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:45.279 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:30:45.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T16:30:45.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:45.279 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:45.978 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:46.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:46.980 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:46.980 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:30:46.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T16:30:46.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:46.981 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:47.393 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:48.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:48.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:48.395 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:30:48.395 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T16:30:48.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:48.396 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:48.940 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:49.941 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:49.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:49.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T16:30:49.942 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:30:49.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:49.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:50.288 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:51.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:51.289 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:51.290 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:30:51.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T16:30:51.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:51.290 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:51.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:52.716 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:52.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:52.717 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T16:30:52.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T16:30:52.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:52.717 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:53.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:30:54.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:30:54.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:30:54.500 INFO:tasks.workunit.client.0.smithi112.stdout:26 2024-09-13T16:30:54.500 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 26 2024-09-13T16:30:54.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:30:54.501 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:30:54.850 INFO:tasks.workunit.client.0.smithi112.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3500408485,v1:127.0.0.1:6811/3500408485] [v2:127.0.0.1:6812/3500408485,v1:127.0.0.1:6813/3500408485] exists,up 2299a6cb-852f-4a5b-a91f-0f7d58220942 2024-09-13T16:30:54.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:30:54.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:30:54.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:30:54.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:46: TEST_recover_unexpected: run_osd td/osd-unexpected-clone 2 2024-09-13T16:30:54.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-unexpected-clone 2024-09-13T16:30:54.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-13T16:30:54.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-13T16:30:54.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-13T16:30:54.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-unexpected-clone/2 2024-09-13T16:30:54.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-13T16:30:54.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:30:54.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:30:54.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:30:54.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2024-09-13T16:30:54.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2024-09-13T16:30:54.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-13T16:30:54.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-13T16:30:54.853 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-13T16:30:54.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-13T16:30:54.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:30:54.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:30:54.855 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:30:54.855 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:30:54.855 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:30:54.855 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:30:54.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:30:54.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:30:54.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-13T16:30:54.855 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-13T16:30:54.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-13T16:30:54.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-13T16:30:54.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:30:54.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:30:54.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:30:54.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:30:54.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-13T16:30:54.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-13T16:30:54.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-unexpected-clone/2 2024-09-13T16:30:54.856 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-13T16:30:54.856 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=aac4148f-3637-4ee1-9880-8e06140f7950 2024-09-13T16:30:54.857 INFO:tasks.workunit.client.0.smithi112.stdout:add osd2 aac4148f-3637-4ee1-9880-8e06140f7950 2024-09-13T16:30:54.857 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 aac4148f-3637-4ee1-9880-8e06140f7950' 2024-09-13T16:30:54.857 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-13T16:30:54.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC+aORmBuwANBAAykXEazAUy8s+WTldFHQmAQ== 2024-09-13T16:30:54.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC+aORmBuwANBAAykXEazAUy8s+WTldFHQmAQ=="}' 2024-09-13T16:30:54.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new aac4148f-3637-4ee1-9880-8e06140f7950 -i td/osd-unexpected-clone/2/new.json 2024-09-13T16:30:55.319 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:30:55.337 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-unexpected-clone/2/new.json 2024-09-13T16:30:55.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC+aORmBuwANBAAykXEazAUy8s+WTldFHQmAQ== --osd-uuid aac4148f-3637-4ee1-9880-8e06140f7950 2024-09-13T16:30:55.371 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:30:55.372+0000 7f64a0f9f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:30:55.463 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:30:55.464+0000 7f64a0f9f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:30:55.465 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:30:55.464+0000 7f64a0f9f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:30:55.465 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:30:55.468+0000 7f64a0f9f7c0 -1 bdev(0x55ae205cfc00 td/osd-unexpected-clone/2/block) open stat got: (1) Operation not permitted 2024-09-13T16:30:55.466 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:30:55.468+0000 7f64a0f9f7c0 -1 bluestore(td/osd-unexpected-clone/2) _read_fsid unparsable uuid 2024-09-13T16:31:04.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-unexpected-clone/2/keyring 2024-09-13T16:31:04.073 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-13T16:31:04.074 INFO:tasks.workunit.client.0.smithi112.stdout:adding osd2 key to auth repository 2024-09-13T16:31:04.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-13T16:31:04.075 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-unexpected-clone/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-13T16:31:04.470 INFO:tasks.workunit.client.0.smithi112.stdout:start osd.2 2024-09-13T16:31:04.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-13T16:31:04.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T16:31:04.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-13T16:31:04.471 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-13T16:31:04.472 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-13T16:31:04.500 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:31:04.499+0000 7f9e3fed67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:31:04.502 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:31:04.503+0000 7f9e3fed67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:31:04.503 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:31:04.503+0000 7f9e3fed67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:31:04.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-13T16:31:04.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:31:04.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T16:31:04.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:31:04.780 INFO:tasks.workunit.client.0.smithi112.stdout:0 2024-09-13T16:31:04.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:31:04.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:04.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:31:04.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:04.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:05.093 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:06.017 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:31:06.015+0000 7f9e3fed67c0 -1 Falling back to public interface 2024-09-13T16:31:06.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:06.095 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:06.095 INFO:tasks.workunit.client.0.smithi112.stdout:1 2024-09-13T16:31:06.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:31:06.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:06.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:06.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:07.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:07.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:07.409 INFO:tasks.workunit.client.0.smithi112.stdout:2 2024-09-13T16:31:07.409 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:31:07.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:07.410 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:07.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:08.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:08.726 INFO:tasks.workunit.client.0.smithi112.stdout:3 2024-09-13T16:31:08.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:08.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:31:08.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:08.727 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:09.000 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:31:08.999+0000 7f9e3fed67c0 -1 osd.2 0 log_to_monitors true 2024-09-13T16:31:09.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:10.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:10.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:10.048 INFO:tasks.workunit.client.0.smithi112.stdout:4 2024-09-13T16:31:10.052 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:31:10.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:10.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:10.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:11.406 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:11.406 INFO:tasks.workunit.client.0.smithi112.stdout:5 2024-09-13T16:31:11.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:11.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-13T16:31:11.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:11.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:11.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:12.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:12.843 INFO:tasks.workunit.client.0.smithi112.stdout:6 2024-09-13T16:31:12.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:12.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-13T16:31:12.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:12.844 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:13.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:14.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:14.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:14.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-13T16:31:14.220 INFO:tasks.workunit.client.0.smithi112.stdout:7 2024-09-13T16:31:14.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:14.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:14.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:15.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:15.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:15.535 INFO:tasks.workunit.client.0.smithi112.stdout:8 2024-09-13T16:31:15.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-13T16:31:15.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:15.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:15.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:16.996 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:16.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:16.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-13T16:31:16.997 INFO:tasks.workunit.client.0.smithi112.stdout:9 2024-09-13T16:31:16.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:16.997 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:17.507 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:18.508 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:18.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:18.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-13T16:31:18.509 INFO:tasks.workunit.client.0.smithi112.stdout:10 2024-09-13T16:31:18.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:18.509 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:18.839 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:19.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:19.841 INFO:tasks.workunit.client.0.smithi112.stdout:11 2024-09-13T16:31:19.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:19.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-13T16:31:19.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:19.842 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:20.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:21.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:21.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:21.319 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-13T16:31:21.319 INFO:tasks.workunit.client.0.smithi112.stdout:12 2024-09-13T16:31:21.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:21.320 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:21.712 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:22.714 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:22.714 INFO:tasks.workunit.client.0.smithi112.stdout:13 2024-09-13T16:31:22.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:22.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-13T16:31:22.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:22.715 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:23.267 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:24.268 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:24.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:24.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-13T16:31:24.269 INFO:tasks.workunit.client.0.smithi112.stdout:14 2024-09-13T16:31:24.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:24.269 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:24.601 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:25.603 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:25.603 INFO:tasks.workunit.client.0.smithi112.stdout:15 2024-09-13T16:31:25.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:25.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-13T16:31:25.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:25.604 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:25.953 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:26.955 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:26.955 INFO:tasks.workunit.client.0.smithi112.stdout:16 2024-09-13T16:31:26.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:26.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-13T16:31:26.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:26.956 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:27.405 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:28.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:28.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:28.407 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-13T16:31:28.407 INFO:tasks.workunit.client.0.smithi112.stdout:17 2024-09-13T16:31:28.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:28.408 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:28.741 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:29.742 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:29.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:29.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-13T16:31:29.743 INFO:tasks.workunit.client.0.smithi112.stdout:18 2024-09-13T16:31:29.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:29.743 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:30.570 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:31.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:31.573 INFO:tasks.workunit.client.0.smithi112.stdout:19 2024-09-13T16:31:31.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:31.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-13T16:31:31.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:31.574 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:32.094 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:33.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:33.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:33.097 INFO:tasks.workunit.client.0.smithi112.stdout:20 2024-09-13T16:31:33.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-13T16:31:33.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:33.097 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:33.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:34.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:34.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:34.580 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-13T16:31:34.580 INFO:tasks.workunit.client.0.smithi112.stdout:21 2024-09-13T16:31:34.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:34.581 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:35.087 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:36.088 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:36.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:36.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 22 2024-09-13T16:31:36.089 INFO:tasks.workunit.client.0.smithi112.stdout:22 2024-09-13T16:31:36.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:36.089 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:36.682 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:37.683 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:37.684 INFO:tasks.workunit.client.0.smithi112.stdout:23 2024-09-13T16:31:37.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:37.684 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 23 2024-09-13T16:31:37.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:37.685 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:38.780 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:39.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:39.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:39.781 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 24 2024-09-13T16:31:39.782 INFO:tasks.workunit.client.0.smithi112.stdout:24 2024-09-13T16:31:39.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:39.782 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:40.830 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:41.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:41.832 INFO:tasks.workunit.client.0.smithi112.stdout:25 2024-09-13T16:31:41.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:41.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 25 2024-09-13T16:31:41.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:41.833 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:42.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:43.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:43.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:43.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 26 2024-09-13T16:31:43.158 INFO:tasks.workunit.client.0.smithi112.stdout:26 2024-09-13T16:31:43.158 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:43.159 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:31:43.496 INFO:tasks.workunit.client.0.smithi112.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2263110971,v1:127.0.0.1:6819/2263110971] [v2:127.0.0.1:6820/2263110971,v1:127.0.0.1:6821/2263110971] exists,up aac4148f-3637-4ee1-9880-8e06140f7950 2024-09-13T16:31:43.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:31:43.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:31:43.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:31:43.497 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:48: TEST_recover_unexpected: ceph osd pool create foo 1 2024-09-13T16:31:44.077 INFO:tasks.workunit.client.0.smithi112.stderr:pool 'foo' created 2024-09-13T16:31:44.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:49: TEST_recover_unexpected: rados -p foo put foo /etc/passwd 2024-09-13T16:31:46.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:50: TEST_recover_unexpected: rados -p foo mksnap snap 2024-09-13T16:31:47.231 INFO:tasks.workunit.client.0.smithi112.stdout:created pool foo snap snap 2024-09-13T16:31:47.235 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:51: TEST_recover_unexpected: rados -p foo put foo /etc/group 2024-09-13T16:31:48.031 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:53: TEST_recover_unexpected: wait_for_clean 2024-09-13T16:31:48.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:31:48.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:31:48.032 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:31:48.033 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:31:48.034 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:31:48.034 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:31:48.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:31:48.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:31:48.034 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:31:48.151 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:31:48.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:31:48.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:31:48.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:31:48.152 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:31:48.152 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:31:48.484 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:31:48.484 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:31:48.484 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:31:48.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:31:48.485 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:31:48.485 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:31:48.629 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836505 2024-09-13T16:31:48.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836505 2024-09-13T16:31:48.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505' 2024-09-13T16:31:48.630 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:31:48.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:31:48.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-13T16:31:48.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-13T16:31:48.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672977' 2024-09-13T16:31:48.770 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:31:48.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:31:48.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509453 2024-09-13T16:31:48.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509453 2024-09-13T16:31:48.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672977 2-64424509453' 2024-09-13T16:31:48.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:31:48.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836505 2024-09-13T16:31:48.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:31:48.915 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:31:48.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836505 2024-09-13T16:31:48.916 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:31:48.917 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836505 2024-09-13T16:31:48.918 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-13T16:31:48.918 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.0 seq 21474836505 2024-09-13T16:31:48.918 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:31:49.242 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836505 2024-09-13T16:31:49.243 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:31:50.244 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:31:50.245 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:31:50.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-13T16:31:50.573 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:31:50.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-13T16:31:50.574 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:31:50.576 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:31:50.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-13T16:31:50.577 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:31:50.578 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-13T16:31:50.579 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.1 seq 42949672977 2024-09-13T16:31:50.579 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-13T16:31:50.579 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:31:50.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672978 -lt 42949672977 2024-09-13T16:31:50.933 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:31:50.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509453 2024-09-13T16:31:50.934 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:31:50.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:31:50.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:31:50.936 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509453 2024-09-13T16:31:50.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509453 2024-09-13T16:31:50.938 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-13T16:31:50.938 INFO:tasks.workunit.client.0.smithi112.stdout:waiting osd.2 seq 64424509453 2024-09-13T16:31:50.938 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:31:51.286 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-13T16:31:51.287 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:31:51.287 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:31:51.288 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:31:51.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:31:51.658 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:31:51.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:31:51.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:31:51.659 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:31:51.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:31:51.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:31:51.660 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:31:51.962 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:31:51.963 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:31:51.963 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:31:51.964 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:31:52.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:31:52.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:31:52.328 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:31:52.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:55: TEST_recover_unexpected: get_primary foo foo 2024-09-13T16:31:52.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=foo 2024-09-13T16:31:52.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=foo 2024-09-13T16:31:52.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map foo foo 2024-09-13T16:31:52.329 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-13T16:31:52.654 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:55: TEST_recover_unexpected: local osd=1 2024-09-13T16:31:52.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:57: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 1 --op list foo 2024-09-13T16:31:52.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2024-09-13T16:31:52.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T16:31:52.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T16:31:52.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T16:31:52.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:57: TEST_recover_unexpected: grep snapid.:1 2024-09-13T16:31:52.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 1 --op list foo 2024-09-13T16:31:52.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2024-09-13T16:31:52.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T16:31:52.657 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T16:31:52.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T16:31:52.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.1 2024-09-13T16:31:52.658 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:31:52.658 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:31:52.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:31:52.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:31:52.658 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:31:53.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:31:53.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 1 --op list foo 2024-09-13T16:31:53.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2024-09-13T16:31:53.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T16:31:53.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T16:31:53.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T16:31:53.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/1 2024-09-13T16:31:53.968 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/1 --op list foo 2024-09-13T16:31:54.803 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 1 2024-09-13T16:31:54.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2024-09-13T16:31:54.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T16:31:54.804 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T16:31:54.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T16:31:54.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/1 2024-09-13T16:31:54.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-13T16:31:54.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:31:54.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:31:54.805 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:31:54.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/1' 2024-09-13T16:31:54.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/1/journal' 2024-09-13T16:31:54.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T16:31:54.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T16:31:54.806 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-13T16:31:54.806 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T16:31:54.806 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:31:54.807 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:31:54.807 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:31:54.807 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:31:54.807 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:31:54.807 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:31:54.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:31:54.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:31:54.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-13T16:31:54.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-13T16:31:54.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:31:54.807 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:31:54.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:31:54.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:31:54.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T16:31:54.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T16:31:54.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/1 2024-09-13T16:31:54.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T16:31:54.808 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T16:31:54.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T16:31:54.808 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/1/whoami 2024-09-13T16:31:54.808 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T16:31:54.809 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T16:31:54.809 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T16:31:54.809 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T16:31:54.835 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:31:54.834+0000 7fd14b80f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:31:54.838 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:31:54.838+0000 7fd14b80f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:31:54.839 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:31:54.838+0000 7fd14b80f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:31:55.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T16:31:55.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:31:55.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T16:31:55.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:31:55.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:31:55.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:55.126 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:31:55.126 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T16:31:55.127 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:55.127 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:31:55.444 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:55.865 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:31:55.862+0000 7fd14b80f7c0 -1 Falling back to public interface 2024-09-13T16:31:56.445 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:56.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:56.446 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:31:56.446 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:31:56.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:56.447 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:31:56.757 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:57.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:57.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:57.759 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:31:57.759 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T16:31:57.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:57.760 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:31:58.085 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:59.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:31:59.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:31:59.087 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:31:59.087 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T16:31:59.088 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:31:59.088 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:31:59.412 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:31:59.661 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:31:59.658+0000 7fd14b80f7c0 -1 osd.1 20 log_to_monitors true 2024-09-13T16:32:00.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:32:00.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:00.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:32:00.414 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T16:32:00.415 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:00.415 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:32:00.798 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 24 up_thru 17 down_at 21 last_clean_interval [10,20) [v2:127.0.0.1:6810/2885642721,v1:127.0.0.1:6811/2885642721] [v2:127.0.0.1:6812/2885642721,v1:127.0.0.1:6813/2885642721] exists,up 2299a6cb-852f-4a5b-a91f-0f7d58220942 2024-09-13T16:32:00.798 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:32:00.798 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:32:00.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:32:00.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T16:32:00.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:32:00.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:32:00.799 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:32:00.799 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:32:00.800 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:32:00.800 INFO:tasks.workunit.client.0.smithi112.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:32:00.800 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:32:00.800 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:32:00.800 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:32:00.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:32:00.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:32:00.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:32:00.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:32:00.901 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:32:00.901 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:32:01.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:32:01.216 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:32:01.216 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:32:01.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:32:01.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:32:01.216 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:32:01.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836508 2024-09-13T16:32:01.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836508 2024-09-13T16:32:01.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508' 2024-09-13T16:32:01.362 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:32:01.362 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:32:01.558 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215106 2024-09-13T16:32:01.558 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215106 2024-09-13T16:32:01.558 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-103079215106' 2024-09-13T16:32:01.558 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:32:01.558 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:32:01.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509456 2024-09-13T16:32:01.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509456 2024-09-13T16:32:01.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-103079215106 2-64424509456' 2024-09-13T16:32:01.707 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:32:01.708 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836508 2024-09-13T16:32:01.708 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:32:01.708 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:32:01.709 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836508 2024-09-13T16:32:01.709 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:32:01.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836508 2024-09-13T16:32:01.710 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-13T16:32:01.710 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836508 2024-09-13T16:32:01.710 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:32:02.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-13T16:32:02.031 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:32:02.031 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-103079215106 2024-09-13T16:32:02.032 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:32:02.033 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:32:02.034 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-103079215106 2024-09-13T16:32:02.034 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:32:02.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215106 2024-09-13T16:32:02.036 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 103079215106' 2024-09-13T16:32:02.036 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 103079215106 2024-09-13T16:32:02.036 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:32:02.373 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215106 -lt 103079215106 2024-09-13T16:32:02.373 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:32:02.374 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:32:02.374 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509456 2024-09-13T16:32:02.374 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:32:02.375 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509456 2024-09-13T16:32:02.375 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:32:02.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509456 2024-09-13T16:32:02.377 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-13T16:32:02.377 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509456 2024-09-13T16:32:02.377 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:32:02.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509456 2024-09-13T16:32:02.681 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:32:03.682 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:32:03.683 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:32:04.024 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509457 -lt 64424509456 2024-09-13T16:32:04.024 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:32:04.025 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:32:04.025 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:32:04.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:32:04.414 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:32:04.414 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:32:04.414 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:32:04.414 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:32:04.414 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:32:04.414 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:32:04.414 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:32:04.736 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T16:32:04.736 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:32:04.737 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:32:04.737 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:32:05.096 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T16:32:05.096 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-13T16:32:05.096 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-13T16:32:05.096 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-13T16:32:05.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T16:32:05.097 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-13T16:32:05.197 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T16:32:05.197 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:32:05.198 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:32:05.198 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:32:05.198 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:32:05.198 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:32:05.198 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:32:05.198 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:32:05.523 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-13T16:32:05.523 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:32:05.524 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:32:05.524 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:32:05.931 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-13T16:32:05.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-13T16:32:05.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-13T16:32:05.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-13T16:32:05.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-13T16:32:05.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-13T16:32:05.932 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-13T16:32:05.932 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-13T16:32:05.932 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-13T16:32:06.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-13T16:32:06.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-13T16:32:06.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-13T16:32:06.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-13T16:32:06.300 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-13T16:32:06.501 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-13T16:32:06.501 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:32:06.502 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:32:06.502 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:32:06.502 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:32:06.502 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:32:06.502 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:32:06.502 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:32:06.827 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:32:06.827 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:32:06.828 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:32:06.828 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:32:07.222 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:32:07.222 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:32:07.222 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:32:07.222 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:57: TEST_recover_unexpected: JSON='["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T16:32:07.223 INFO:tasks.workunit.client.0.smithi112.stdout:JSON is ["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}] 2024-09-13T16:32:07.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:58: TEST_recover_unexpected: echo 'JSON is ["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' 2024-09-13T16:32:07.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:59: TEST_recover_unexpected: rm -f td/osd-unexpected-clone/_ td/osd-unexpected-clone/data 2024-09-13T16:32:07.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:60: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2024-09-13T16:32:07.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2024-09-13T16:32:07.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T16:32:07.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-13T16:32:07.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T16:32:07.224 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2024-09-13T16:32:07.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2024-09-13T16:32:07.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T16:32:07.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-13T16:32:07.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T16:32:07.225 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.1 2024-09-13T16:32:07.226 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:32:07.226 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:32:07.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:32:07.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:32:07.226 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:32:08.535 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:32:08.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2024-09-13T16:32:08.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2024-09-13T16:32:08.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T16:32:08.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-13T16:32:08.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T16:32:08.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/1 2024-09-13T16:32:08.536 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2024-09-13T16:32:11.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 1 2024-09-13T16:32:11.116 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2024-09-13T16:32:11.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T16:32:11.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-13T16:32:11.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T16:32:11.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/1 2024-09-13T16:32:11.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-13T16:32:11.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:32:11.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:32:11.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:32:11.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/1' 2024-09-13T16:32:11.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/1/journal' 2024-09-13T16:32:11.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T16:32:11.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T16:32:11.117 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-13T16:32:11.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T16:32:11.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:32:11.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:32:11.118 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:32:11.118 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:32:11.118 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:32:11.118 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:32:11.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:32:11.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:32:11.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-13T16:32:11.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-13T16:32:11.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:32:11.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:32:11.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:32:11.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:32:11.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T16:32:11.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T16:32:11.120 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/1 2024-09-13T16:32:11.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-13T16:32:11.121 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.1 2024-09-13T16:32:11.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T16:32:11.121 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/1/whoami 2024-09-13T16:32:11.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-13T16:32:11.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T16:32:11.121 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T16:32:11.122 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T16:32:11.150 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:11.149+0000 7f0f7e2957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:32:11.153 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:11.149+0000 7f0f7e2957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:32:11.154 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:11.153+0000 7f0f7e2957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:32:11.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-13T16:32:11.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:32:11.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-13T16:32:11.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:32:11.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:32:11.455 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:11.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:32:11.456 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T16:32:11.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:11.456 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:32:11.776 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:32:12.441 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:12.437+0000 7f0f7e2957c0 -1 Falling back to public interface 2024-09-13T16:32:12.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:32:12.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:12.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:32:12.778 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:32:12.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:12.779 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:32:13.096 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:32:14.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:32:14.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:14.098 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:32:14.099 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T16:32:14.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:14.099 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:32:14.420 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:32:15.309 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:15.305+0000 7f0f7e2957c0 -1 osd.1 25 log_to_monitors true 2024-09-13T16:32:15.421 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:32:15.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:15.422 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:32:15.422 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T16:32:15.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:15.423 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:32:15.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:32:16.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:32:16.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:16.826 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:32:16.827 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T16:32:16.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:16.827 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-13T16:32:17.156 INFO:tasks.workunit.client.0.smithi112.stderr:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6810/1872720535,v1:127.0.0.1:6811/1872720535] [v2:127.0.0.1:6812/1872720535,v1:127.0.0.1:6813/1872720535] exists,up 2299a6cb-852f-4a5b-a91f-0f7d58220942 2024-09-13T16:32:17.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:32:17.156 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:32:17.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:32:17.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T16:32:17.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:32:17.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:32:17.157 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:32:17.157 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:32:17.158 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:32:17.158 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:32:17.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:32:17.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:32:17.158 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:32:17.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:32:17.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:32:17.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:32:17.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:32:17.256 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:32:17.256 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:32:17.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:32:17.592 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:32:17.592 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:32:17.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:32:17.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:32:17.593 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:32:17.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836513 2024-09-13T16:32:17.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836513 2024-09-13T16:32:17.764 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513' 2024-09-13T16:32:17.765 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:32:17.765 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:32:17.894 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051586 2024-09-13T16:32:17.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051586 2024-09-13T16:32:17.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-124554051586' 2024-09-13T16:32:17.895 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:32:17.895 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:32:18.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509461 2024-09-13T16:32:18.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509461 2024-09-13T16:32:18.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-124554051586 2-64424509461' 2024-09-13T16:32:18.033 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:32:18.033 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836513 2024-09-13T16:32:18.033 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:32:18.034 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:32:18.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836513 2024-09-13T16:32:18.035 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:32:18.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836513 2024-09-13T16:32:18.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-13T16:32:18.036 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836513 2024-09-13T16:32:18.036 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:32:18.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836513 2024-09-13T16:32:18.338 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:32:19.339 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:32:19.340 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:32:19.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836513 2024-09-13T16:32:19.676 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:32:20.677 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-13T16:32:20.678 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:32:20.988 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836513 -lt 21474836513 2024-09-13T16:32:20.989 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:32:20.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:32:20.989 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-124554051586 2024-09-13T16:32:20.990 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:32:20.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-124554051586 2024-09-13T16:32:20.991 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:32:20.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051586 2024-09-13T16:32:20.992 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-09-13T16:32:20.992 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 124554051586 2024-09-13T16:32:20.993 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:32:21.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-13T16:32:21.297 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:32:21.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509461 2024-09-13T16:32:21.298 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:32:21.299 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:32:21.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509461 2024-09-13T16:32:21.299 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:32:21.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509461 2024-09-13T16:32:21.301 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-13T16:32:21.301 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 64424509461 2024-09-13T16:32:21.301 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:32:21.669 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-13T16:32:21.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:32:21.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:32:21.670 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:32:22.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:32:22.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:32:22.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:32:22.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:32:22.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:32:22.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:32:22.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:32:22.048 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:32:22.355 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:32:22.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:32:22.355 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:32:22.356 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:32:22.724 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:32:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:32:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:32:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:61: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2024-09-13T16:32:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2024-09-13T16:32:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T16:32:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-13T16:32:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T16:32:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2024-09-13T16:32:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2024-09-13T16:32:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T16:32:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-13T16:32:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T16:32:22.725 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.2 2024-09-13T16:32:22.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:32:22.726 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:32:22.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:32:22.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:32:22.726 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:32:24.035 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:32:24.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2024-09-13T16:32:24.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2024-09-13T16:32:24.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T16:32:24.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-13T16:32:24.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T16:32:24.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/2 2024-09-13T16:32:24.036 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2024-09-13T16:32:24.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 2 2024-09-13T16:32:24.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2024-09-13T16:32:24.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T16:32:24.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T16:32:24.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T16:32:24.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/2 2024-09-13T16:32:24.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-13T16:32:24.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:32:24.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:32:24.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:32:24.831 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2024-09-13T16:32:24.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2024-09-13T16:32:24.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T16:32:24.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T16:32:24.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-13T16:32:24.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T16:32:24.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:32:24.832 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:32:24.833 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:32:24.834 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:32:24.834 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:32:24.834 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:32:24.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:32:24.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:32:24.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-13T16:32:24.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-13T16:32:24.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:32:24.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:32:24.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:32:24.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:32:24.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T16:32:24.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T16:32:24.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/2 2024-09-13T16:32:24.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T16:32:24.835 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.2 2024-09-13T16:32:24.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T16:32:24.835 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/2/whoami 2024-09-13T16:32:24.836 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T16:32:24.837 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T16:32:24.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T16:32:24.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T16:32:24.867 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:24.865+0000 7f11f0f1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:32:24.869 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:24.865+0000 7f11f0f1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:32:24.871 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:24.869+0000 7f11f0f1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:32:25.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T16:32:25.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:32:25.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T16:32:25.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:32:25.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:32:25.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:25.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:32:25.173 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T16:32:25.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:25.173 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:32:25.496 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:32:26.145 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:26.141+0000 7f11f0f1f7c0 -1 Falling back to public interface 2024-09-13T16:32:26.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:32:26.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:26.498 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:32:26.498 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:32:26.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:26.499 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:32:26.832 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:32:27.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:32:27.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:27.834 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:32:27.834 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T16:32:27.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:27.835 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:32:28.203 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:32:29.205 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:32:29.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:29.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:32:29.206 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T16:32:29.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:29.206 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:32:29.527 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:32:29.815 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:29.813+0000 7f11f0f1f7c0 -1 osd.2 30 log_to_monitors true 2024-09-13T16:32:30.529 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:32:30.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:30.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:32:30.530 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T16:32:30.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:30.530 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:32:30.889 INFO:tasks.workunit.client.0.smithi112.stderr:osd.2 up in weight 1 up_from 34 up_thru 0 down_at 31 last_clean_interval [15,30) [v2:127.0.0.1:6818/2163000564,v1:127.0.0.1:6819/2163000564] [v2:127.0.0.1:6820/2163000564,v1:127.0.0.1:6821/2163000564] exists,up aac4148f-3637-4ee1-9880-8e06140f7950 2024-09-13T16:32:30.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:32:30.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:32:30.889 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:32:30.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T16:32:30.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:32:30.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:32:30.890 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:32:30.890 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:32:30.890 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:32:30.891 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:32:30.891 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:32:30.891 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:32:30.891 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:32:30.984 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:32:30.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:32:30.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:32:30.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:32:30.985 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:32:30.985 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:32:31.317 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:32:31.317 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:32:31.318 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:32:31.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:32:31.318 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:32:31.318 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:32:31.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836516 2024-09-13T16:32:31.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836516 2024-09-13T16:32:31.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516' 2024-09-13T16:32:31.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:32:31.467 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:32:31.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051590 2024-09-13T16:32:31.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051590 2024-09-13T16:32:31.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516 1-124554051590' 2024-09-13T16:32:31.671 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:32:31.672 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:32:31.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888066 2024-09-13T16:32:31.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888066 2024-09-13T16:32:31.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516 1-124554051590 2-146028888066' 2024-09-13T16:32:31.851 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:32:31.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836516 2024-09-13T16:32:31.852 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:32:31.852 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:32:31.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836516 2024-09-13T16:32:31.853 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:32:31.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836516 2024-09-13T16:32:31.854 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-09-13T16:32:31.854 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836516 2024-09-13T16:32:31.854 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:32:32.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836516 -lt 21474836516 2024-09-13T16:32:32.216 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:32:32.216 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-124554051590 2024-09-13T16:32:32.217 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:32:32.217 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:32:32.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-124554051590 2024-09-13T16:32:32.218 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:32:32.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051590 2024-09-13T16:32:32.219 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 124554051590' 2024-09-13T16:32:32.219 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 124554051590 2024-09-13T16:32:32.219 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:32:32.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051589 -lt 124554051590 2024-09-13T16:32:32.585 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:32:33.587 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:32:33.587 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:32:33.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051590 -lt 124554051590 2024-09-13T16:32:33.926 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:32:33.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-146028888066 2024-09-13T16:32:33.927 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:32:33.928 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:32:33.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-146028888066 2024-09-13T16:32:33.929 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:32:33.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888066 2024-09-13T16:32:33.931 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 146028888066' 2024-09-13T16:32:33.931 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 146028888066 2024-09-13T16:32:33.931 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:32:34.238 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888066 -lt 146028888066 2024-09-13T16:32:34.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:32:34.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:32:34.239 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:32:34.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:32:34.628 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:32:34.629 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:32:34.629 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:32:34.629 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:32:34.629 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:32:34.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:32:34.630 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:32:34.994 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:32:34.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:32:34.994 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:32:34.995 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:32:35.384 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:32:35.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:32:35.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:32:35.385 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:63: TEST_recover_unexpected: rados -p foo rmsnap snap 2024-09-13T16:32:35.747 INFO:tasks.workunit.client.0.smithi112.stdout:removed pool foo snap snap 2024-09-13T16:32:35.750 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:65: TEST_recover_unexpected: sleep 5 2024-09-13T16:32:40.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:67: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2024-09-13T16:32:40.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2024-09-13T16:32:40.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T16:32:40.753 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-13T16:32:40.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T16:32:40.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2024-09-13T16:32:40.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2024-09-13T16:32:40.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T16:32:40.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-13T16:32:40.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T16:32:40.754 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.2 2024-09-13T16:32:40.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:32:40.755 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:32:40.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:32:40.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:32:40.755 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:32:41.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:32:41.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2024-09-13T16:32:41.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2024-09-13T16:32:41.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T16:32:41.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-13T16:32:41.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T16:32:41.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/2 2024-09-13T16:32:41.066 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2024-09-13T16:32:43.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 2 2024-09-13T16:32:43.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2024-09-13T16:32:43.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T16:32:43.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T16:32:43.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T16:32:43.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/2 2024-09-13T16:32:43.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-13T16:32:43.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:32:43.814 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:32:43.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:32:43.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2024-09-13T16:32:43.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2024-09-13T16:32:43.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T16:32:43.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T16:32:43.815 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-13T16:32:43.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T16:32:43.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:32:43.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:32:43.815 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:32:43.815 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:32:43.815 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:32:43.815 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:32:43.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:32:43.816 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:32:43.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-13T16:32:43.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-13T16:32:43.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:32:43.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:32:43.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:32:43.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:32:43.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T16:32:43.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T16:32:43.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/2 2024-09-13T16:32:43.817 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T16:32:43.817 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.2 2024-09-13T16:32:43.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T16:32:43.818 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/2/whoami 2024-09-13T16:32:43.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T16:32:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T16:32:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T16:32:43.819 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T16:32:43.849 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:43.845+0000 7f925b5a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:32:43.851 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:43.849+0000 7f925b5a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:32:43.853 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:43.849+0000 7f925b5a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:32:44.149 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T16:32:44.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:32:44.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T16:32:44.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:32:44.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:32:44.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:44.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:32:44.150 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T16:32:44.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:44.150 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:32:44.521 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:32:45.329 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:45.325+0000 7f925b5a17c0 -1 Falling back to public interface 2024-09-13T16:32:45.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:32:45.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:45.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:32:45.523 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:32:45.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:45.524 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:32:45.867 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:32:46.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:32:46.869 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:46.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:32:46.870 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T16:32:46.870 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:46.871 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:32:47.186 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:32:48.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:32:48.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:48.188 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:32:48.188 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T16:32:48.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:48.189 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:32:48.569 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:32:48.629 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:48.624+0000 7f925b5a17c0 -1 osd.2 36 log_to_monitors true 2024-09-13T16:32:49.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:32:49.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:49.571 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:32:49.571 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T16:32:49.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:49.572 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:32:49.913 INFO:tasks.workunit.client.0.smithi112.stderr:osd.2 up in weight 1 up_from 41 up_thru 0 down_at 37 last_clean_interval [34,36) [v2:127.0.0.1:6818/1563130475,v1:127.0.0.1:6819/1563130475] [v2:127.0.0.1:6820/1563130475,v1:127.0.0.1:6821/1563130475] exists,up aac4148f-3637-4ee1-9880-8e06140f7950 2024-09-13T16:32:49.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:32:49.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:32:49.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:32:49.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T16:32:49.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:32:49.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:32:49.914 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:32:49.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:32:49.915 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:32:49.915 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:32:49.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:32:49.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:32:49.915 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:32:50.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:32:50.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:32:50.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:32:50.021 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:32:50.022 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:32:50.022 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:32:50.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:32:50.348 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:32:50.348 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:32:50.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:32:50.348 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:32:50.348 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:32:50.623 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836521 2024-09-13T16:32:50.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836521 2024-09-13T16:32:50.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521' 2024-09-13T16:32:50.624 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:32:50.624 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:32:50.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051595 2024-09-13T16:32:50.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051595 2024-09-13T16:32:50.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-124554051595' 2024-09-13T16:32:50.801 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:32:50.801 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:32:50.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659138 2024-09-13T16:32:50.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659138 2024-09-13T16:32:50.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-124554051595 2-176093659138' 2024-09-13T16:32:50.942 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:32:50.943 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836521 2024-09-13T16:32:50.943 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:32:50.944 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:32:50.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836521 2024-09-13T16:32:50.945 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:32:50.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836521 2024-09-13T16:32:50.946 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-09-13T16:32:50.946 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836521 2024-09-13T16:32:50.946 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:32:51.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836520 -lt 21474836521 2024-09-13T16:32:51.258 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:32:52.259 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:32:52.260 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:32:52.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836521 -lt 21474836521 2024-09-13T16:32:52.588 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:32:52.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-124554051595 2024-09-13T16:32:52.589 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:32:52.590 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:32:52.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-124554051595 2024-09-13T16:32:52.590 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:32:52.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051595 2024-09-13T16:32:52.592 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 124554051595' 2024-09-13T16:32:52.592 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 124554051595 2024-09-13T16:32:52.592 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:32:52.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051595 -lt 124554051595 2024-09-13T16:32:52.969 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:32:52.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-176093659138 2024-09-13T16:32:52.970 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:32:52.971 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:32:52.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-176093659138 2024-09-13T16:32:52.972 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:32:52.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659138 2024-09-13T16:32:52.973 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 176093659138' 2024-09-13T16:32:52.973 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 176093659138 2024-09-13T16:32:52.973 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:32:53.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659138 -lt 176093659138 2024-09-13T16:32:53.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:32:53.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:32:53.352 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:32:53.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:32:53.769 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:32:53.769 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:32:53.769 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:32:53.769 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:32:53.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:32:53.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:32:53.770 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:32:54.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:32:54.075 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:32:54.075 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:32:54.075 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:32:54.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:32:54.466 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:32:54.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:32:54.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:68: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2024-09-13T16:32:54.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2024-09-13T16:32:54.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-13T16:32:54.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-13T16:32:54.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-13T16:32:54.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2024-09-13T16:32:54.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2024-09-13T16:32:54.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-13T16:32:54.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-13T16:32:54.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-13T16:32:54.467 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.2 2024-09-13T16:32:54.467 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:32:54.468 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:32:54.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:32:54.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:32:54.468 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:32:55.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:32:55.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2024-09-13T16:32:55.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2024-09-13T16:32:55.777 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-13T16:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-13T16:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-13T16:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/2 2024-09-13T16:32:55.778 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2024-09-13T16:32:58.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 2 2024-09-13T16:32:58.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2024-09-13T16:32:58.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-13T16:32:58.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-13T16:32:58.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-13T16:32:58.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/2 2024-09-13T16:32:58.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 ' 2024-09-13T16:32:58.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-13T16:32:58.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-13T16:32:58.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-13T16:32:58.209 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2024-09-13T16:32:58.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2024-09-13T16:32:58.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-13T16:32:58.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-13T16:32:58.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2024-09-13T16:32:58.211 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-13T16:32:58.211 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-13T16:32:58.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-13T16:32:58.212 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-13T16:32:58.212 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:32:58.212 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:32:58.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:32:58.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' 2024-09-13T16:32:58.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-13T16:32:58.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2024-09-13T16:32:58.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2024-09-13T16:32:58.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-13T16:32:58.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-13T16:32:58.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-13T16:32:58.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-13T16:32:58.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-13T16:32:58.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-13T16:32:58.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/2 2024-09-13T16:32:58.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-13T16:32:58.214 INFO:tasks.workunit.client.0.smithi112.stderr:start osd.2 2024-09-13T16:32:58.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=10872ed6-e96c-46ba-9cd3-b788dfc2b7b0 --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.643418/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-13T16:32:58.214 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/2/whoami 2024-09-13T16:32:58.214 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-13T16:32:58.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-13T16:32:58.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-13T16:32:58.215 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-13T16:32:58.251 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:58.248+0000 7fa6f51447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:32:58.254 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:58.252+0000 7fa6f51447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:32:58.255 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:58.252+0000 7fa6f51447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-13T16:32:58.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-13T16:32:58.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-13T16:32:58.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-13T16:32:58.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-13T16:32:58.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-13T16:32:58.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:58.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-13T16:32:58.533 INFO:tasks.workunit.client.0.smithi112.stderr:0 2024-09-13T16:32:58.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:58.533 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:32:58.901 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:32:59.301 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:32:59.300+0000 7fa6f51447c0 -1 Falling back to public interface 2024-09-13T16:32:59.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:32:59.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:32:59.903 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-13T16:32:59.904 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:32:59.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:32:59.904 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:33:00.218 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:33:01.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:33:01.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:33:01.220 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-13T16:33:01.220 INFO:tasks.workunit.client.0.smithi112.stderr:2 2024-09-13T16:33:01.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:33:01.221 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:33:01.546 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:33:02.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:33:02.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:33:02.548 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-13T16:33:02.548 INFO:tasks.workunit.client.0.smithi112.stderr:3 2024-09-13T16:33:02.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:33:02.549 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:33:02.586 INFO:tasks.workunit.client.0.smithi112.stderr:2024-09-13T16:33:02.584+0000 7fa6f51447c0 -1 osd.2 42 log_to_monitors true 2024-09-13T16:33:02.885 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-13T16:33:03.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-13T16:33:03.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-13T16:33:03.886 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-13T16:33:03.886 INFO:tasks.workunit.client.0.smithi112.stderr:4 2024-09-13T16:33:03.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-13T16:33:03.887 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-13T16:33:04.250 INFO:tasks.workunit.client.0.smithi112.stderr:osd.2 up in weight 1 up_from 46 up_thru 0 down_at 43 last_clean_interval [41,42) [v2:127.0.0.1:6818/3012203830,v1:127.0.0.1:6819/3012203830] [v2:127.0.0.1:6820/3012203830,v1:127.0.0.1:6821/3012203830] exists,up aac4148f-3637-4ee1-9880-8e06140f7950 2024-09-13T16:33:04.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-13T16:33:04.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-13T16:33:04.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-13T16:33:04.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-13T16:33:04.250 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-13T16:33:04.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-13T16:33:04.251 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-13T16:33:04.251 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-13T16:33:04.251 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-13T16:33:04.252 INFO:tasks.workunit.client.0.smithi112.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-13T16:33:04.252 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-13T16:33:04.252 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-13T16:33:04.252 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-13T16:33:04.350 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-13T16:33:04.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-13T16:33:04.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-13T16:33:04.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-13T16:33:04.351 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-13T16:33:04.351 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-13T16:33:04.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-13T16:33:04.744 INFO:tasks.workunit.client.0.smithi112.stderr:1 2024-09-13T16:33:04.744 INFO:tasks.workunit.client.0.smithi112.stderr:2' 2024-09-13T16:33:04.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-13T16:33:04.744 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:33:04.744 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-13T16:33:04.934 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836525 2024-09-13T16:33:04.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836525 2024-09-13T16:33:04.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525' 2024-09-13T16:33:04.935 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:33:04.935 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-13T16:33:05.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051598 2024-09-13T16:33:05.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051598 2024-09-13T16:33:05.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525 1-124554051598' 2024-09-13T16:33:05.074 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-13T16:33:05.075 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-13T16:33:05.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=197568495618 2024-09-13T16:33:05.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 197568495618 2024-09-13T16:33:05.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836525 1-124554051598 2-197568495618' 2024-09-13T16:33:05.210 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:33:05.210 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836525 2024-09-13T16:33:05.211 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:33:05.211 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-13T16:33:05.211 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836525 2024-09-13T16:33:05.212 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:33:05.212 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836525 2024-09-13T16:33:05.213 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-09-13T16:33:05.213 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.0 seq 21474836525 2024-09-13T16:33:05.213 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:33:05.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836525 2024-09-13T16:33:05.522 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-13T16:33:06.523 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-13T16:33:06.524 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-13T16:33:06.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836525 -lt 21474836525 2024-09-13T16:33:06.838 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:33:06.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-124554051598 2024-09-13T16:33:06.839 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:33:06.841 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-13T16:33:06.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-124554051598 2024-09-13T16:33:06.841 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:33:06.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051598 2024-09-13T16:33:06.843 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 124554051598' 2024-09-13T16:33:06.843 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.1 seq 124554051598 2024-09-13T16:33:06.843 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-13T16:33:07.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051598 -lt 124554051598 2024-09-13T16:33:07.136 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-13T16:33:07.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-197568495618 2024-09-13T16:33:07.136 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-13T16:33:07.138 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-13T16:33:07.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-197568495618 2024-09-13T16:33:07.139 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-13T16:33:07.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=197568495618 2024-09-13T16:33:07.140 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 197568495618' 2024-09-13T16:33:07.140 INFO:tasks.workunit.client.0.smithi112.stderr:waiting osd.2 seq 197568495618 2024-09-13T16:33:07.140 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-13T16:33:07.450 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 197568495618 -lt 197568495618 2024-09-13T16:33:07.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-13T16:33:07.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:33:07.451 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:33:07.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-13T16:33:07.824 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-13T16:33:07.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-13T16:33:07.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-13T16:33:07.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-13T16:33:07.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-13T16:33:07.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-13T16:33:07.825 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-13T16:33:08.146 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-13T16:33:08.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-13T16:33:08.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-13T16:33:08.146 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-13T16:33:08.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-13T16:33:08.503 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-13T16:33:08.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-13T16:33:08.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:70: TEST_recover_unexpected: sleep 5 2024-09-13T16:33:13.504 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:72: TEST_recover_unexpected: ceph pg repair 1.0 2024-09-13T16:33:13.802 INFO:tasks.workunit.client.0.smithi112.stderr:instructing pg 1.0 on osd.1 to repair 2024-09-13T16:33:13.818 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:74: TEST_recover_unexpected: sleep 10 2024-09-13T16:33:23.820 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:76: TEST_recover_unexpected: ceph log last 2024-09-13T16:33:24.192 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T16:33:14.458169+0000 osd.1 (osd.1) 3 : cluster [ERR] 1.0 shard 1 1:602f83fe:::foo:1 : missing 2024-09-13T16:33:24.192 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T16:33:14.458411+0000 osd.1 (osd.1) 4 : cluster [ERR] repair 1.0 1:602f83fe:::foo:1 : is an unexpected clone 2024-09-13T16:33:24.192 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T16:33:14.458694+0000 osd.1 (osd.1) 5 : cluster [ERR] 1.0 repair 1 missing, 0 inconsistent objects 2024-09-13T16:33:24.192 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T16:33:14.458727+0000 osd.1 (osd.1) 6 : cluster [ERR] 1.0 repair 1 missing, 0 inconsistent objects 2024-09-13T16:33:24.192 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T16:33:14.458781+0000 osd.1 (osd.1) 7 : cluster [ERR] 1.0 repair 3 errors, 2 fixed 2024-09-13T16:33:24.193 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T16:33:15.467457+0000 osd.1 (osd.1) 9 : cluster [ERR] 1.0 shard 1 soid 1:602f83fe:::foo:1 : candidate size 0 info size 13101 mismatch 2024-09-13T16:33:24.193 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T16:33:15.467621+0000 osd.1 (osd.1) 10 : cluster [ERR] 1.0 shard 0 soid 1:602f83fe:::foo:1 : data_digest 0xd6c424ed != data_digest 0x9a3eb952 from auth oi 1:602f83fe:::foo:1(20'2 client.4298.0:1 dirty|data_digest s 13101 uv 1 dd 9a3eb952 alloc_hint [0 0 0]) 2024-09-13T16:33:24.193 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T16:33:15.467626+0000 osd.1 (osd.1) 11 : cluster [ERR] 1.0 shard 1 soid 1:602f83fe:::foo:1 : data_digest 0xffffffff != data_digest 0xd6c424ed from shard 0, data_digest 0xffffffff != data_digest 0x9a3eb952 from auth oi 1:602f83fe:::foo:1(20'2 client.4298.0:1 dirty|data_digest s 13101 uv 1 dd 9a3eb952 alloc_hint [0 0 0]), size 0 != size 13101 from auth oi 1:602f83fe:::foo:1(20'2 client.4298.0:1 dirty|data_digest s 13101 uv 1 dd 9a3eb952 alloc_hint [0 0 0]), size 0 != size 13101 from shard 0 2024-09-13T16:33:24.193 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T16:33:15.467628+0000 osd.1 (osd.1) 12 : cluster [ERR] 1.0 soid 1:602f83fe:::foo:1 : data_digest 0x9a3eb952 != data_digest 0xd6c424ed from shard 0 2024-09-13T16:33:24.193 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T16:33:15.467866+0000 osd.1 (osd.1) 13 : cluster [ERR] deep-scrub 1.0 1:602f83fe:::foo:1 : is an unexpected clone 2024-09-13T16:33:24.193 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T16:33:15.468238+0000 osd.1 (osd.1) 14 : cluster [ERR] 1.0 deep-scrub 0 missing, 1 inconsistent objects 2024-09-13T16:33:24.193 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T16:33:15.468244+0000 osd.1 (osd.1) 15 : cluster [ERR] 1.0 deep-scrub 4 errors 2024-09-13T16:33:24.193 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T16:33:17.633351+0000 mon.a (mon.0) 428 : cluster [ERR] Health check failed: 4 scrub errors (OSD_SCRUB_ERRORS) 2024-09-13T16:33:24.193 INFO:tasks.workunit.client.0.smithi112.stdout:2024-09-13T16:33:17.633408+0000 mon.a (mon.0) 429 : cluster [ERR] Health check failed: Possible data damage: 1 pg inconsistent (PG_DAMAGED) 2024-09-13T16:33:24.207 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:79: TEST_recover_unexpected: timeout 60 ceph tell osd.0 version 2024-09-13T16:33:24.387 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:33:24.388 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "19.3.0-4697-gd58f7306", 2024-09-13T16:33:24.388 INFO:tasks.workunit.client.0.smithi112.stdout: "release": "squid", 2024-09-13T16:33:24.388 INFO:tasks.workunit.client.0.smithi112.stdout: "release_type": "dev" 2024-09-13T16:33:24.388 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:33:24.404 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:80: TEST_recover_unexpected: timeout 60 ceph tell osd.1 version 2024-09-13T16:33:24.539 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:33:24.539 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "19.3.0-4697-gd58f7306", 2024-09-13T16:33:24.539 INFO:tasks.workunit.client.0.smithi112.stdout: "release": "squid", 2024-09-13T16:33:24.539 INFO:tasks.workunit.client.0.smithi112.stdout: "release_type": "dev" 2024-09-13T16:33:24.539 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:33:24.554 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:81: TEST_recover_unexpected: timeout 60 ceph tell osd.2 version 2024-09-13T16:33:24.684 INFO:tasks.workunit.client.0.smithi112.stdout:{ 2024-09-13T16:33:24.684 INFO:tasks.workunit.client.0.smithi112.stdout: "version": "19.3.0-4697-gd58f7306", 2024-09-13T16:33:24.684 INFO:tasks.workunit.client.0.smithi112.stdout: "release": "squid", 2024-09-13T16:33:24.684 INFO:tasks.workunit.client.0.smithi112.stdout: "release_type": "dev" 2024-09-13T16:33:24.684 INFO:tasks.workunit.client.0.smithi112.stdout:} 2024-09-13T16:33:24.698 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:35: run: teardown td/osd-unexpected-clone 2024-09-13T16:33:24.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-unexpected-clone 2024-09-13T16:33:24.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-13T16:33:24.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-unexpected-clone KILL 2024-09-13T16:33:24.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:33:24.699 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:33:24.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:33:24.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:33:24.699 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:33:25.010 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:33:25.010 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:33:25.011 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:33:25.012 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:33:25.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:33:25.014 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:33:25.014 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:33:25.015 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:33:25.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:33:25.016 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:33:25.017 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:33:25.018 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:33:25.019 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-13T16:33:25.020 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-unexpected-clone 2024-09-13T16:33:25.044 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:33:25.044 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:33:25.044 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:33:25.044 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.643418 2024-09-13T16:33:25.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:33:25.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:33:25.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-13T16:33:25.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-unexpected-clone 0 2024-09-13T16:33:25.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-unexpected-clone 2024-09-13T16:33:25.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-13T16:33:25.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-unexpected-clone KILL 2024-09-13T16:33:25.046 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-13T16:33:25.046 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-13T16:33:25.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-13T16:33:25.046 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-13T16:33:25.047 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-13T16:33:25.048 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-13T16:33:25.049 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-13T16:33:25.049 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-13T16:33:25.050 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-13T16:33:25.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-13T16:33:25.051 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-13T16:33:25.052 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-13T16:33:25.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:33:25.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-13T16:33:25.053 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-13T16:33:25.054 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:33:25.054 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:33:25.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-13T16:33:25.055 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-unexpected-clone 2024-09-13T16:33:25.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-13T16:33:25.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-13T16:33:25.057 INFO:tasks.workunit.client.0.smithi112.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.643418 2024-09-13T16:33:25.057 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.643418 2024-09-13T16:33:25.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-13T16:33:25.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-13T16:33:25.059 INFO:tasks.workunit.client.0.smithi112.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-13T16:33:25.059 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T16:33:25.060 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T16:33:25.114 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-09-13T16:33:25.114 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-13T16:33:25.467 DEBUG:teuthology.parallel:result is None 2024-09-13T16:33:25.467 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T16:33:25.477 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T16:33:25.478 DEBUG:teuthology.orchestra.run.smithi112:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:33:25.542 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T16:33:25.542 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T16:33:25.553 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T16:33:25.554 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T16:33:25.633 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T16:33:25.640 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T16:33:25.641 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T16:33:25.657 INFO:teuthology.orchestra.run.smithi112.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T16:33:25.658 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================== 2024-09-13T16:33:25.658 INFO:teuthology.orchestra.run.smithi112.stdout:*hv01.front.sepi 67.205.162.81 3 u 186 256 377 0.156 -1.878 0.783 2024-09-13T16:33:25.658 INFO:teuthology.orchestra.run.smithi112.stdout:+hv02.front.sepi 63.231.80.2 3 u 83 256 377 0.160 -2.795 1.081 2024-09-13T16:33:25.658 INFO:teuthology.orchestra.run.smithi112.stdout:+hv03.front.sepi 74.6.168.72 3 u 80 256 377 0.186 -2.550 0.462 2024-09-13T16:33:25.658 INFO:teuthology.orchestra.run.smithi112.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-13T16:33:25.659 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T16:33:25.667 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T16:33:25.668 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T16:33:25.694 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T16:33:25.729 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T16:33:25.761 INFO:teuthology.task.internal:Duration was 9995.087798 seconds 2024-09-13T16:33:25.762 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T16:33:25.795 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T16:33:25.795 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T16:33:25.840 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T16:33:25.840 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2024-09-13T16:33:25.840 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T16:33:25.892 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T16:33:25.892 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T16:33:25.971 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T16:33:25.972 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T16:33:26.076 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T16:33:26.086 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T16:33:26.086 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T16:33:26.099 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T16:33:26.133 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:33:26.153 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2024-09-13T16:33:26.167 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:33:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:33:26.214 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T16:33:26.223 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T16:33:26.225 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/akupczyk-2024-09-13_09:43:51-rados-wip-aclamk-cbt-combined-distro-default-smithi/7903521/remote/smithi112 2024-09-13T16:33:26.225 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T16:33:26.283 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T16:33:26.283 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T16:33:26.326 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T16:33:26.338 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T16:33:26.339 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T16:33:26.362 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T16:33:26.362 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T16:33:26.374 INFO:teuthology.orchestra.run.smithi112.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 16:33 /home/ubuntu/cephtest 2024-09-13T16:33:26.376 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T16:33:26.396 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T16:33:26.465 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} duration: 9995.087797641754 flavor: default owner: scheduled_akupczyk@teuthology success: true 2024-09-13T16:33:26.465 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T16:33:26.546 INFO:teuthology.run:pass